update on book

doc/src/DimRed/chapter8.dlog

@@ -0,0 +1,75 @@
+*** error: file has a mako construction ${\mathbb{R}'
+    but seemingly no definition in <%...%>'
+    (it is not a command-line given mako variable either).
+    However, if this is a variable in a Makefile or Bash script
+    run with --no_mako - and you cannot use mako and Makefile or Bash variables
+    in the same document!
+
+*** error: file has a mako construction ${\mathbb{R}'
+    but seemingly no definition in <%...%>'
+    (it is not a command-line given mako variable either).
+    However, if this is a variable in a Makefile or Bash script
+    run with --no_mako - and you cannot use mako and Makefile or Bash variables
+    in the same document!
+
+translating doconce text in chapter8.do.txt to ipynb
+*** replacing \bm{...} by \boldsymbol{...} (\bm is not supported by MathJax)
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+Failed to remove ans_at_end environment
+Failed to remove sol_at_end environment
+output in chapter8.ipynb

doc/src/How2ReadData/chapter3.dlog

@@ -0,0 +1,22 @@
+translating doconce text in chapter3.do.txt to ipynb
+*** replacing \bm{...} by \boldsymbol{...} (\bm is not supported by MathJax)
+collected all required additional files in ipynb-chapter3-src.tar.gz which must be distributed with the notebook
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+*** warning: latex envir \begin{cases} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+Failed to remove ans_at_end environment
+Failed to remove sol_at_end environment
+output in chapter3.ipynb

doc/src/LectureNotes/_build/html/chapter1.html

@@ -174,6 +174,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/chapter2.html

@@ -175,6 +175,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>
@@ -1001,27 +1011,27 @@ uncorrelated.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>1.5027679581515354
-[[ 4.01839868  4.38511101  9.23577076  8.53622276  4.78693205  9.01249172
-   5.62006613  2.52380443  6.68351121  4.12787844]
- [ 4.38511101  4.78528891 10.07861174  9.31522416  5.22377946  9.83495664
-   6.13294396  2.7541226   7.29343725  4.50458172]
- [ 9.23577076 10.07861174 21.22722716 19.61940636 11.00214554 20.71404911
-  12.9169967   5.80063878 15.36118796  9.48739587]
- [ 8.53622276  9.31522416 19.61940636 18.13336726 10.16880644 19.14509813
-  11.9386204   5.3612791  14.19768049  8.76878895]
- [ 4.78693205  5.22377946 11.00214554 10.16880644  5.70245024 10.73616358
-   6.69492424  3.00649122  7.96175706  4.91735022]
- [ 9.01249172  9.83495664 20.71404911 19.14509813 10.73616358 20.21327739
-  12.60472232  5.66040565 14.98982413  9.25803368]
- [ 5.62006613  6.13294396 12.9169967  11.9386204   6.69492424 12.60472232
-   7.86013182  3.52975127  9.34744857  5.77318272]
- [ 2.52380443  2.7541226   5.80063878  5.3612791   3.00649122  5.66040565
-   3.52975127  1.58510624  4.19766095  2.59256454]
- [ 6.68351121  7.29343725 15.36118796 14.19768049  7.96175706 14.98982413
-   9.34744857  4.19766095 11.11619967  6.86560096]
- [ 4.12787844  4.50458172  9.48739587  8.76878895  4.91735022  9.25803368
-   5.77318272  2.59256454  6.86560096  4.24034094]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>9.106279149747735
+[[  1.87336856   1.77787047   1.44771535   6.44982327  11.43684153
+    5.83694909   1.84228806   7.37557482  -2.58314202   6.48643762]
+ [  1.77787047   1.68724056   1.37391564   6.12103277  10.85382947
+    5.53940088   1.74837435   6.9995926   -2.45146205   6.15578065]
+ [  1.44771535   1.37391564   1.11877597   4.98434123   8.83824539
+    4.51071987   1.42369673   5.69975023  -1.99621924   5.01263633]
+ [  6.44982327   6.12103277   4.98434123  22.2061057   39.37591798
+   20.09604032   6.34281617  25.39337703  -8.89350334  22.33216531]
+ [ 11.43684153  10.85382947   8.83824539  39.37591798  69.82146882
+   35.63434516  11.2470963   45.02759486 -15.76998069  39.59944717]
+ [  5.83694909   5.53940088   4.51071987  20.09604032  35.63434516
+   18.18647726   5.74011     22.9804512   -8.04842615  20.21012151]
+ [  1.84228806   1.74837435   1.42369673   6.34281617  11.2470963
+    5.74011      1.8117232    7.25320885  -2.54028588   6.37882306]
+ [  7.37557482   6.9995926    5.69975023  25.39337703  45.02759486
+   22.9804512    7.25320885  29.03812156 -10.16999948  25.53753014]
+ [ -2.58314202  -2.45146205  -1.99621924  -8.89350334 -15.76998069
+   -8.04842615  -2.54028588 -10.16999948   3.56183127  -8.94398998]
+ [  6.48643762   6.15578065   5.01263633  22.33216531  39.59944717
+   20.21012151   6.37882306  25.53753014  -8.94398998  22.45894054]]
 </pre></div>
 </div>
 </div>
@@ -1329,15 +1339,15 @@ more practically oriented methods like the blocking technique.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.02709116808235872
-4.0262891855828515
--0.05083752107727271
-1.1034925742304618 9.662232920789233 19.396576628082066
-3.1026745228913506 3.57651731449913 9.947888130154134
-[[ 1.10349257  3.10267452  3.57651731]
- [ 3.10267452  9.66223292  9.94788813]
- [ 3.57651731  9.94788813 19.39657663]]
-[26.46728898  0.08444587  3.61056727]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.06894012083823547
+4.139781119573823
+0.04321525078901231
+1.0006918520539008 10.50426403458282 18.88724031810869
+3.0717579594084814 3.4645963550106305 10.40024628605229
+[[ 1.00069185  3.07175796  3.46459636]
+ [ 3.07175796 10.50426403 10.40024629]
+ [ 3.46459636 10.40024629 18.88724032]]
+[26.72833966  0.07676079  3.58709575]
 </pre></div>
 </div>
 </div>
@@ -1744,7 +1754,7 @@ assumption for approximating <span class="math notranslate nohighlight">\(\sigma
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.02793163482104733 1.025167138849349
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.027481252820017347 1.0170183171235068
 </pre></div>
 </div>
 <img alt="_images/chapter2_184_1.png" src="_images/chapter2_184_1.png" />

doc/src/LectureNotes/_build/html/chapter3.html

@@ -175,6 +175,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>
@@ -700,8 +710,8 @@ developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly he
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[-0.80600218 -0.30092    -0.79536928  0.14039618  0.5768749   0.74732035
- -2.28459617 -0.84483144 -1.24760167  1.04875861]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 0.99499832 -0.89728339 -1.69744895 -1.03875025 -0.07638981  0.18716123
+ -0.27804028  0.72149922  1.25862131 -0.7970463 ]
 </pre></div>
 </div>
 </div>

doc/src/LectureNotes/_build/html/chapter4.html

@@ -175,6 +175,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>
@@ -1806,13 +1816,13 @@ but now splitting the data into a training set and a test set.</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Training R2
-0.9999853406074647
+0.9999886705644145
 Training MSE
-6.359080163429899
+3.90943518299982
 Test R2
-0.9999859407754539
+0.9999697792755088
 Test MSE
-6.980914000813206
+25.32441051671905
 </pre></div>
 </div>
 </div>
@@ -1943,7 +1953,7 @@ dtype: int64
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.axes._subplots.AxesSubplot at 0x7fddb4c436d0&gt;
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.axes._subplots.AxesSubplot at 0x7fb5be1db040&gt;
 </pre></div>
 </div>
 <img alt="_images/chapter4_131_1.png" src="_images/chapter4_131_1.png" />
@@ -2229,27 +2239,27 @@ techniques.</p>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>MSE before scaling: 0.00
 R2 score before scaling 0.99
 Feature min values before scaling:
- [1.00000000e+00 4.28119384e-04 8.30273573e-04 1.83286207e-07
- 3.55456211e-07 6.89354206e-07 7.84683780e-11 1.52177694e-10
- 2.95125898e-10 5.72352580e-10 3.35938336e-14 6.51502206e-14
- 1.26349118e-13 2.45035234e-13 4.75209222e-13 1.43821714e-17
- 2.78920723e-17 5.40925064e-17 1.04904333e-16 2.03446279e-16
- 3.94553659e-16]
+ [1.00000000e+00 1.97624658e-03 6.76071445e-04 3.90555053e-06
+ 1.33608388e-06 4.57072598e-07 7.71833086e-09 2.64043119e-09
+ 9.03288157e-10 3.09013732e-10 1.52533249e-11 5.21814310e-12
+ 1.78512013e-12 6.10687329e-13 2.08915360e-13 3.01443312e-14
+ 1.03123374e-14 3.52783754e-15 1.20686874e-15 4.12868265e-16
+ 1.41241709e-16]
 Feature max values before scaling:
- [1.         0.99959919 0.99554286 0.99919855 0.99514384 0.99110558
- 0.99879806 0.99474497 0.99070834 0.98668808 0.99839773 0.99434627
- 0.99031125 0.98629261 0.98229027 0.99799757 0.99394773 0.98991433
- 0.98589729 0.98189656 0.97791206]
+ [1.         0.99729116 0.99990303 0.99458965 0.99719445 0.99980607
+ 0.99189546 0.9944932  0.99709775 0.99970911 0.98920857 0.99179928
+ 0.99439677 0.99700106 0.99961217 0.98652896 0.98911265 0.9917031
+ 0.99430034 0.99690438 0.99951524]
 Feature min values after scaling:
- [ 0.         -1.69931545 -1.67469653 -1.11879252 -1.10594061 -1.09316288
- -0.88278129 -0.87818569 -0.8737743  -0.8695324  -0.74532693 -0.74499485
- -0.74480651 -0.74474355 -0.7447873  -0.65291273 -0.65458157 -0.65636186
- -0.6582414  -0.6602078  -0.66224856]
+ [ 0.         -1.72006556 -1.76752166 -1.10773734 -1.11611153 -1.12503152
+ -0.87804483 -0.88146268 -0.88496565 -0.8885653  -0.75100539 -0.75305518
+ -0.75511928 -0.75719979 -0.75929911 -0.66720819 -0.66863343 -0.67005915
+ -0.67148589 -0.67291428 -0.67434497]
 Feature max values after scaling:
- [0.         1.73355301 1.72505955 2.27111673 2.24946337 2.22510326
- 2.69980853 2.67945831 2.65749797 2.63400218 3.05843173 3.04339787
- 3.02720478 3.00984683 2.99131512 3.36770789 3.3585133  3.3484316
- 3.33743441 3.32549161 3.31257165]
+ [0.         1.74774217 1.73485006 2.2506647  2.23491553 2.21893993
+ 2.6752011  2.65930564 2.64314172 2.62671501 3.05237641 3.03669249
+ 3.02075875 3.00457804 2.98815333 3.39653406 3.38082542 3.36489022
+ 3.34873108 3.33235061 3.31575145]
 MSE after  scaling: 0.00
 R2 score for  scaled data: 0.99
 </pre></div>
@@ -2839,10 +2849,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.10594950732957698
-4.624126522020202
-[[ 0.994928    3.03386068]
- [ 3.03386068 10.13854864]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.12208685625303164
+4.452659449239899
+[[ 1.00910422  3.12769989]
+ [ 3.12769989 10.63920917]]
 </pre></div>
 </div>
 </div>
@@ -2882,10 +2892,10 @@ a more brute force way. Here we scale the mean values for each column of the des
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07844342450165018
-1.1438626259785865
-[[1.         0.60716876]
- [0.60716876 1.        ]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08487462066865184
+1.7716882265595972
+[[1.         0.74332853]
+ [0.74332853 1.        ]]
 </pre></div>
 </div>
 </div>
@@ -2917,30 +2927,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-0.1066151   0.79747251]
- [ 0.75209514  1.96762409]
- [-0.41638994 -2.34035396]
- [-0.2780316  -1.49418072]
- [ 0.86865915  2.72245363]
- [ 0.20418073  0.86260647]
- [-0.79048758 -0.42464144]
- [-0.01768994 -0.1467412 ]
- [-0.26355349 -1.81771904]
- [ 0.04783264 -0.12652035]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-0.63821798 -2.03548189]
+ [ 0.98355854  2.40965456]
+ [ 0.48870683  2.68995497]
+ [ 0.44655566  1.28908336]
+ [ 0.3871261  -0.67155367]
+ [-0.32256574 -0.55849157]
+ [ 1.3507663   2.75066843]
+ [-1.44489727 -5.37462032]
+ [-0.42087991  1.26840089]
+ [-0.83015254 -1.76761477]]
           0         1
-0 -0.106615  0.797473
-1  0.752095  1.967624
-2 -0.416390 -2.340354
-3 -0.278032 -1.494181
-4  0.868659  2.722454
-5  0.204181  0.862606
-6 -0.790488 -0.424641
-7 -0.017690 -0.146741
-8 -0.263553 -1.817719
-9  0.047833 -0.126520
+0 -0.638218 -2.035482
+1  0.983559  2.409655
+2  0.488707  2.689955
+3  0.446556  1.289083
+4  0.387126 -0.671554
+5 -0.322566 -0.558492
+6  1.350766  2.750668
+7 -1.444897 -5.374620
+8 -0.420880  1.268401
+9 -0.830153 -1.767615
           0         1
-0  1.000000  0.824095
-1  0.824095  1.000000
+0  1.000000  0.877156
+1  0.877156  1.000000
 </pre></div>
 </div>
 </div>
@@ -3000,37 +3010,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>     0         1         2         3         4         5         6         7   \
 0   0.0  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   
-1   0.0  0.090154  0.084830  0.094410  0.086610  0.079643  0.088034  0.080631   
-2   0.0  0.084830  0.081616  0.091708  0.085377  0.079558  0.087469  0.081068   
-3   0.0  0.094410  0.091708  0.104704  0.097816  0.091403  0.101195  0.093913   
-4   0.0  0.086610  0.085377  0.097816  0.092341  0.087099  0.095895  0.089763   
-5   0.0  0.079643  0.079558  0.091403  0.087099  0.082837  0.090734  0.085584   
-6   0.0  0.088034  0.087469  0.101195  0.095895  0.090734  0.100269  0.094057   
-7   0.0  0.080631  0.081068  0.093913  0.089763  0.085584  0.094057  0.088862   
-8   0.0  0.074145  0.075363  0.087388  0.084186  0.080823  0.088367  0.084031   
-9   0.0  0.068449  0.070272  0.081540  0.079119  0.076438  0.083169  0.079557   
-10  0.0  0.080692  0.081580  0.095076  0.091141  0.087107  0.095945  0.090806   
-11  0.0  0.074184  0.075759  0.088306  0.085279  0.082038  0.089878  0.085590   
-12  0.0  0.068486  0.070593  0.082283  0.080006  0.077425  0.084399  0.080829   
-13  0.0  0.063479  0.065996  0.076913  0.075254  0.073225  0.079446  0.076483   
-14  0.0  0.059064  0.061893  0.072112  0.070966  0.069398  0.074962  0.072512   
+1   0.0  0.085618  0.079643  0.085857  0.084047  0.081888  0.076531  0.075718   
+2   0.0  0.079643  0.075359  0.078265  0.077130  0.075774  0.069143  0.068658   
+3   0.0  0.085857  0.078265  0.090778  0.088194  0.085138  0.083648  0.082461   
+4   0.0  0.084047  0.077130  0.088194  0.085938  0.083258  0.080985  0.079981   
+5   0.0  0.081888  0.075774  0.085138  0.083258  0.081011  0.077856  0.077054   
+6   0.0  0.076531  0.069143  0.083648  0.080985  0.077856  0.078889  0.077645   
+7   0.0  0.075718  0.068658  0.082461  0.079981  0.077054  0.077645  0.076512   
+8   0.0  0.074845  0.068154  0.081161  0.078881  0.076175  0.076275  0.075261   
+9   0.0  0.073877  0.067608  0.079704  0.077645  0.075189  0.074736  0.073852   
+10  0.0  0.067084  0.060409  0.075015  0.072536  0.069633  0.071991  0.070821   
+11  0.0  0.066572  0.060088  0.074312  0.071948  0.069168  0.071268  0.070173   
+12  0.0  0.066065  0.059784  0.073591  0.071348  0.068699  0.070516  0.069500   
+13  0.0  0.065552  0.059492  0.072838  0.070724  0.068215  0.069723  0.068789   
+14  0.0  0.065022  0.059205  0.072037  0.070061  0.067706  0.068871  0.068023   
 
           8         9         10        11        12        13        14  
 0   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  
-1   0.074145  0.068449  0.080692  0.074184  0.068486  0.063479  0.059064  
-2   0.075363  0.070272  0.081580  0.075759  0.070593  0.065996  0.061893  
-3   0.087388  0.081540  0.095076  0.088306  0.082283  0.076913  0.072112  
-4   0.084186  0.079119  0.091141  0.085279  0.080006  0.075254  0.070966  
-5   0.080823  0.076438  0.087107  0.082038  0.077425  0.073225  0.069398  
-6   0.088367  0.083169  0.095945  0.089878  0.084399  0.079446  0.074962  
-7   0.084031  0.079557  0.090806  0.085590  0.080829  0.076483  0.072512  
-8   0.079929  0.076076  0.085994  0.081509  0.077368  0.073550  0.070030  
-9   0.076076  0.072757  0.081515  0.077656  0.074052  0.070694  0.067570  
-10  0.085994  0.081515  0.093107  0.087851  0.083035  0.078625  0.074586  
-11  0.081509  0.077656  0.087851  0.083337  0.079156  0.075290  0.071718  
-12  0.077368  0.074052  0.083035  0.079156  0.075521  0.072127  0.068962  
-13  0.073550  0.070694  0.078625  0.075290  0.072127  0.069142  0.066334  
-14  0.070030  0.067570  0.074586  0.071718  0.068962  0.066334  0.063837  
+1   0.074845  0.073877  0.067084  0.066572  0.066065  0.065552  0.065022  
+2   0.068154  0.067608  0.060409  0.060088  0.059784  0.059492  0.059205  
+3   0.081161  0.079704  0.075015  0.074312  0.073591  0.072838  0.072037  
+4   0.078881  0.077645  0.072536  0.071948  0.071348  0.070724  0.070061  
+5   0.076175  0.075189  0.069633  0.069168  0.068699  0.068215  0.067706  
+6   0.076275  0.074736  0.071991  0.071268  0.070516  0.069723  0.068871  
+7   0.075261  0.073852  0.070821  0.070173  0.069500  0.068789  0.068023  
+8   0.074139  0.072869  0.069531  0.068962  0.068371  0.067746  0.067072  
+9   0.072869  0.071752  0.068081  0.067595  0.067092  0.066559  0.065985  
+10  0.069531  0.068081  0.066617  0.065944  0.065240  0.064493  0.063686  
+11  0.068962  0.067595  0.065944  0.065324  0.064676  0.063985  0.063237  
+12  0.068371  0.067092  0.065240  0.064676  0.064083  0.063452  0.062765  
+13  0.067746  0.066559  0.064493  0.063985  0.063452  0.062881  0.062258  
+14  0.067072  0.065985  0.063686  0.063237  0.062765  0.062258  0.061702  
 </pre></div>
 </div>
 </div>
@@ -3380,10 +3390,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.425629 sec
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.430516 sec
 Jackknife Statistics :
 original           bias      std. error
-   99.99          99.98        0.151321
+ 100.186        100.176        0.153008
 </pre></div>
 </div>
 </div>
@@ -3525,10 +3535,10 @@ theorem.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 2.1542 sec
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 2.20285 sec
 Bootstrap Statistics :
 original           bias      std. error
- 99.8081  15.0184        99.8106        0.150604
+ 100.167   14.919        100.169        0.150847
 </pre></div>
 </div>
 <div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>

doc/src/LectureNotes/_build/html/chapter5.html

@@ -175,6 +175,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/chapter6.html

@@ -38,6 +38,7 @@
     <script async="async" src="_static/sphinx-thebe.js"></script>
     <link rel="index" title="Index" href="genindex.html" />
     <link rel="search" title="Search" href="search.html" />
+    <link rel="next" title="19. Support Vector Machines, overarching aims" href="chapter7.html" />
     <link rel="prev" title="17. Logistic Regression" href="chapter5.html" />
 
     <meta name="viewport" content="width=device-width, initial-scale=1">
@@ -174,6 +175,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>
@@ -3223,6 +3234,7 @@ features).</p>
     <div class='prev-next-bottom'>
         
     <a class='left-prev' id="prev-link" href="chapter5.html" title="previous page"><span class="section-number">17. </span>Logistic Regression</a>
+    <a class='right-next' id="next-link" href="chapter7.html" title="next page"><span class="section-number">19. </span>Support Vector Machines, overarching aims</a>
 
     </div>
     <footer class="footer mt-5 mt-md-0">

doc/src/LectureNotes/_build/html/genindex.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/reports/chapter7.log

@@ -0,0 +1,41 @@
+Traceback (most recent call last):
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution
+    executenb(
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 1082, in execute
+    return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute()
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
+    return just_run(coro(*args, **kwargs))
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
+    return loop.run_until_complete(coro)
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
+    return future.result()
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 535, in async_execute
+    await self.async_execute_cell(
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 827, in async_execute_cell
+    self._check_raise_for_error(cell, exec_reply)
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 735, in _check_raise_for_error
+    raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
+nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
+------------------
+# Import the necessary packages
+import numpy
+from cvxopt import matrix
+from cvxopt import solvers
+P = matrix(numpy.diag([1,0]), tc=’d’)
+q = matrix(numpy.array([3,4]), tc=’d’)
+G = matrix(numpy.array([[-1,0],[0,-1],[-1,-3],[2,5],[3,4]]), tc=’d’)
+h = matrix(numpy.array([0,0,-15,100,80]), tc=’d’)
+# Construct the QP, invoke solver
+sol = solvers.qp(P,q,G,h)
+# Extract optimal value and solution
+sol[’x’] 
+sol[’primal objective’]
+------------------
+
+  File "<ipython-input-5-c46dd114b2af>", line 5
+    P = matrix(numpy.diag([1,0]), tc=’d’)
+                                       ^
+SyntaxError: invalid character in identifier
+
+SyntaxError: invalid character in identifier (<ipython-input-5-c46dd114b2af>, line 5)
+

doc/src/LectureNotes/_build/html/reports/chapter8.log

@@ -0,0 +1,29 @@
+Traceback (most recent call last):
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution
+    executenb(
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 1082, in execute
+    return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute()
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
+    return just_run(coro(*args, **kwargs))
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
+    return loop.run_until_complete(coro)
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
+    return future.result()
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 535, in async_execute
+    await self.async_execute_cell(
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 827, in async_execute_cell
+    self._check_raise_for_error(cell, exec_reply)
+  File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 735, in _check_raise_for_error
+    raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
+nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
+------------------
+pca.components_.T[:, 0].
+------------------
+
+  File "<ipython-input-19-17314f270d45>", line 1
+    pca.components_.T[:, 0].
+                            ^
+SyntaxError: invalid syntax
+
+SyntaxError: invalid syntax (<ipython-input-19-17314f270d45>, line 1)
+

doc/src/LectureNotes/_build/html/search.html

@@ -178,6 +178,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/_build/jupyter_execute/chapter1.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/_build/jupyter_execute/chapter2.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>
@@ -1603,8 +1613,8 @@ developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly he
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 0.90559229 -2.01023012 -0.44771747  1.75725059 -1.23503845  0.00290572
- -1.00282695  1.34507057  1.8031745  -0.99146414]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.54613374 -0.53808134 -0.25599148 -2.19980623 -1.26614367  2.09110254
+  0.54374681  0.6362131   1.04152939 -1.69246995]
 </pre></div>
 </div>
 </div>

doc/src/LectureNotes/_build/html/testbook/_build/jupyter_execute/chapter3.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/_build/jupyter_execute/chapter4.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/_build/jupyter_execute/chapter5.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/_build/jupyter_execute/chapter6.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/_build/jupyter_execute/chapter7.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/_build/jupyter_execute/lecturenotes/lecturenotes/notebooks.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/_build/jupyter_execute/notebooks.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/chapter1.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/chapter2.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>
@@ -1603,8 +1613,8 @@ developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly he
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[-1.36647401  0.48392582  1.3866607  -1.32437748 -1.27886869 -0.45735097
-  2.09942022  0.4864173  -0.46526198  1.4136186 ]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 0.37402521 -0.6723554   0.34705159 -0.23106244  0.37640289  1.26261376
+ -1.28899002  0.59279401 -1.28878405 -0.82161508]
 </pre></div>
 </div>
 </div>

doc/src/LectureNotes/_build/html/testbook/chapter3.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/chapter4.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/chapter5.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/chapter6.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/chapter7.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/content.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/intro.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/lecturenotes/CONDUCT.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/lecturenotes/CONTRIBUTING.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/lecturenotes/README.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/lecturenotes/lecturenotes/content.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/lecturenotes/lecturenotes/intro.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/lecturenotes/lecturenotes/markdown.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/lecturenotes/lecturenotes/notebooks.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../../../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/html/testbook/markdown.html

@@ -173,6 +173,16 @@
    18. Neural networks, from the simple perceptron to deep learning
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter7.html">
+   19. Support Vector Machines, overarching aims
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="../chapter8.html">
+   20. Dimensionality Reduction
+  </a>
+ </li>
 </ul>
 
 </nav>

doc/src/LectureNotes/_build/jupyter_execute/chapter1.ipynb

@@ -413,6 +413,8 @@
     "chapter4.ipynb\n",
     "chapter5.ipynb\n",
     "chapter6.ipynb\n",
+    "chapter7.ipynb\n",
+    "chapter8.ipynb\n",
     "```\n"
    ]
   }

doc/src/LectureNotes/_build/jupyter_execute/chapter1.txt

@@ -402,4 +402,6 @@ chapter3.ipynb
 chapter4.ipynb
 chapter5.ipynb
 chapter6.ipynb
+chapter7.ipynb
+chapter8.ipynb
 ```

doc/src/LectureNotes/_build/jupyter_execute/chapter3.ipynb

@@ -440,8 +440,8 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "[-0.80600218 -0.30092    -0.79536928  0.14039618  0.5768749   0.74732035\n",
-      " -2.28459617 -0.84483144 -1.24760167  1.04875861]\n"
+      "[ 0.99499832 -0.89728339 -1.69744895 -1.03875025 -0.07638981  0.18716123\n",
+      " -0.27804028  0.72149922  1.25862131 -0.7970463 ]\n"
      ]
     }
    ],

doc/src/LectureNotes/_build/jupyter_execute/testbook/_build/jupyter_execute/chapter2.ipynb

@@ -2152,8 +2152,8 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "[ 0.90559229 -2.01023012 -0.44771747  1.75725059 -1.23503845  0.00290572\n",
-      " -1.00282695  1.34507057  1.8031745  -0.99146414]\n"
+      "[ 1.54613374 -0.53808134 -0.25599148 -2.19980623 -1.26614367  2.09110254\n",
+      "  0.54374681  0.6362131   1.04152939 -1.69246995]\n"
      ]
     }
    ],

doc/src/LectureNotes/_build/jupyter_execute/testbook/chapter2.ipynb

@@ -2152,8 +2152,8 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "[-1.36647401  0.48392582  1.3866607  -1.32437748 -1.27886869 -0.45735097\n",
-      "  2.09942022  0.4864173  -0.46526198  1.4136186 ]\n"
+      "[ 0.37402521 -0.6723554   0.34705159 -0.23106244  0.37640289  1.26261376\n",
+      " -1.28899002  0.59279401 -1.28878405 -0.82161508]\n"
      ]
     }
    ],

doc/src/LectureNotes/_toc.yml

@@ -7,4 +7,6 @@
   - file: chapter4.ipynb
   - file: chapter5.ipynb
   - file: chapter6.ipynb
+  - file: chapter7.ipynb
+  - file: chapter8.ipynb  
 

doc/src/LogisticRegression/chapter5.dlog

@@ -0,0 +1,8 @@
+translating doconce text in chapter5.do.txt to ipynb
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.
+    Stick to \[ ... \], equation, equation*, align, or align*
+    environments in math environments.
+
+Failed to remove ans_at_end environment
+Failed to remove sol_at_end environment
+output in chapter5.ipynb

doc/src/LogisticRegression/chapter5.do.txt

@@ -0,0 +1,366 @@
+======= Logistic Regression =======
+
+===== Introduction =====
+In linear regression our main interest was centered on learning the
+coefficients of a functional fit (say a polynomial) in order to be
+able to predict the response of a continuous variable on some unseen
+data. The fit to the continuous variable $y_i$ is based on some
+independent variables $\hat{x}_i$. Linear regression resulted in
+analytical expressions for standard ordinary Least Squares or Ridge
+regression (in terms of matrices to invert) for several quantities,
+ranging from the variance and thereby the confidence intervals of the
+parameters $\hat{\beta}$ to the mean squared error. If we can invert
+the product of the design matrices, linear regression gives then a
+simple recipe for fitting our data.
+
+
+Classification problems, however, are concerned with outcomes taking
+the form of discrete variables (i.e. categories). We may for example,
+on the basis of DNA sequencing for a number of patients, like to find
+out which mutations are important for a certain disease; or based on
+scans of various patients' brains, figure out if there is a tumor or
+not; or given a specific physical system, we'd like to identify its
+state, say whether it is an ordered or disordered system (typical
+situation in solid state physics); or classify the status of a
+patient, whether she/he has a stroke or not and many other similar
+situations.
+
+The most common situation we encounter when we apply logistic
+regression is that of two possible outcomes, normally denoted as a
+binary outcome, true or false, positive or negative, success or
+failure etc.
+
+Logistic regression will also serve as our stepping stone towards
+neural network algorithms and supervised deep learning. For logistic
+learning, the minimization of the cost function leads to a non-linear
+equation in the parameters $\hat{\beta}$. The optimization of the
+problem calls therefore for minimization algorithms. This forms the
+bottle neck of all machine learning algorithms, namely how to find
+reliable minima of a multi-variable function. This leads us to the
+family of gradient descent methods. The latter are the working horses
+of basically all modern machine learning algorithms.
+
+We note also that many of the topics discussed here on logistic 
+regression are also commonly used in modern supervised Deep Learning
+models, as we will see later.
+
+
+
+===== Basics =====
+
+We consider the case where the dependent variables, also called the
+responses or the outcomes, $y_i$ are discrete and only take values
+from $k=0,\dots,K-1$ (i.e. $K$ classes).
+
+The goal is to predict the
+output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$
+made of $n$ samples, each of which carries $p$ features or predictors. The
+primary goal is to identify the classes to which new unseen samples
+belong.
+ 
+Let us specialize to the case of two classes only, with outputs
+$y_i=0$ and $y_i=1$. Our outcomes could represent the status of a
+credit card user that could default or not on her/his credit card
+debt. That is
+
+
+!bt
+\[
+y_i = \begin{bmatrix} 0 & \mathrm{no}\\  1 & \mathrm{yes} \end{bmatrix}.
+\]
+!et
+
+
+
+Before moving to the logistic model, let us try to use our linear
+regression model to classify these two outcomes. We could for example
+fit a linear model to the default case if $y_i > 0.5$ and the no
+default case $y_i \leq 0.5$.
+
+We would then have our 
+weighted linear combination, namely 
+!bt
+\begin{equation}
+\hat{y} = \hat{X}^T\hat{\beta} +  \hat{\epsilon},
+\end{equation}
+!et
+where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our
+$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors.
+
+
+The main problem with our function is that it takes values on the
+entire real axis. In the case of logistic regression, however, the
+labels $y_i$ are discrete variables. A typical example is the credit
+card data discussed below here, where we can set the state of
+defaulting the debt to $y_i=1$ and not to $y_i=0$ for one the persons
+in the data set (see the full example below).
+
+One simple way to get a discrete output is to have sign
+functions that map the output of a linear regressor to values $\{0,1\}$,
+$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise. 
+We will encounter this model in our first demonstration of neural networks. Historically it is called the ``perceptron" model in the machine learning
+literature. This model is extremely simple. However, in many cases it is more
+favorable to use a ``soft" classifier that outputs
+the probability of a given category. This leads us to the logistic function.
+
+
+
+===== The logistic function =====
+
+The perceptron is an example of a ``hard classification'' model. We
+will encounter this model when we discuss neural networks as
+well. Each datapoint is deterministically assigned to a category (i.e
+$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a ``soft''
+classifier that outputs the probability of a given category rather
+than a single value. For example, given $x_i$, the classifier
+outputs the probability of being in a category $k$.  Logistic regression
+is the most common example of a so-called soft classifier. In logistic
+regression, the probability that a data point $x_i$
+belongs to a category $y_i=\{0,1\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event, 
+!bt
+\[
+p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
+\]
+!et
+Note that $1-p(t)= p(-t)$.
+
+
+The following code plots the logistic function, the step function and other functions we will encounter from here and on.
+
+
+!bc pycod
+"""The sigmoid function (or the logistic curve) is a
+function that takes any real number, z, and outputs a number (0,1).
+It is useful in neural networks for assigning weights on a relative scale.
+The value z is the weighted sum of parameters involved in the learning algorithm."""
+
+import numpy
+import matplotlib.pyplot as plt
+import math as mt
+
+z = numpy.arange(-5, 5, .1)
+sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
+sigma = sigma_fn(z)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, sigma)
+ax.set_ylim([-0.1, 1.1])
+ax.set_xlim([-5,5])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('sigmoid function')
+
+plt.show()
+
+"""Step Function"""
+z = numpy.arange(-5, 5, .02)
+step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
+step = step_fn(z)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, step)
+ax.set_ylim([-0.5, 1.5])
+ax.set_xlim([-5,5])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('step function')
+
+plt.show()
+
+"""tanh Function"""
+z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
+t = numpy.tanh(z)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, t)
+ax.set_ylim([-1.0, 1.0])
+ax.set_xlim([-2*mt.pi,2*mt.pi])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('tanh function')
+
+plt.show()
+!ec
+
+
+=====  Two parameters =====
+
+We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities 
+!bt
+\begin{align*}
+p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
+p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
+\end{align*}
+!et
+where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. 
+
+Note that we used
+!bt
+\[
+p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
+\]
+!et
+
+===== Maximum likelihood =====
+
+In order to define the total likelihood for all possible outcomes from a  
+dataset $\mathcal{D}=\{(y_i,x_i)\}$, with the binary labels
+$y_i\in\{0,1\}$ and where the data points are drawn independently, we use the so-called "Maximum Likelihood Estimation":"https://en.wikipedia.org/wiki/Maximum_likelihood_estimation" (MLE) principle. 
+We aim thus at maximizing 
+the probability of seeing the observed data. We can then approximate the 
+likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is 
+!bt
+\begin{align*}
+P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
+\end{align*}
+!et
+from which we obtain the log-likelihood and our _cost/loss_ function
+!bt
+\[
+\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
+\]
+!et
+
+
+Reordering the logarithms, we can rewrite the _cost/loss_ function as
+!bt
+\[
+\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
+\]
+!et
+
+The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\beta$.
+Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
+!bt
+\[
+\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
+\]
+!et
+This equation is known in statistics as the _cross entropy_. Finally, we note that just as in linear regression, 
+in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.
+
+
+The cross entropy is a convex function of the weights $\hat{\beta}$ and,
+therefore, any local minimizer is a global minimizer. 
+
+
+Minimizing this
+cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obtain
+
+!bt
+\[
+\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
+\]
+!et
+and 
+!bt
+\[
+\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
+\]
+!et
+
+Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an
+$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a
+vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first
+derivative of cost function as
+
+!bt
+\[
+\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). 
+\]
+!et
+
+If we in addition define a diagonal matrix $\hat{W}$ with elements 
+$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as 
+
+!bt
+\[
+\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. 
+\]
+!et
+
+
+Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors
+!bt
+\[
+\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
+\]
+!et
+Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to
+!bt
+\[
+p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
+\]
+!et
+
+
+===== Including more classes =====
+
+Till now we have mainly focused on two classes, the so-called binary
+system. Suppose we wish to extend to $K$ classes.  Let us for the sake
+of simplicity assume we have only two predictors. We have then
+following model
+
+!bt
+\[
+\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
+\]
+!et
+!bt
+\[
+\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
+\]
+!et
+and so on till the class $C=K-1$ class
+!bt
+\[
+\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
+\]
+!et
+
+and the model is specified in term of $K-1$ so-called log-odds or
+_logit_ transformations.
+
+
+
+In our discussion of neural networks we will encounter the above again
+in terms of a slightly modified function, the so-called _Softmax_ function.
+
+The softmax function is used in various multiclass classification
+methods, such as multinomial logistic regression (also known as
+softmax regression), multiclass linear discriminant analysis, naive
+Bayes classifiers, and artificial neural networks.  Specifically, in
+multinomial logistic regression and linear discriminant analysis, the
+input to the function is the result of $K$ distinct linear functions,
+and the predicted probability for the $k$-th class given a sample
+vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two
+predictors):
+
+!bt
+\[
+p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
+\]
+!et
+It is easy to extend to more predictors. The final class is 
+!bt
+\[
+p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
+\]
+!et
+
+and they sum to one. Our earlier discussions were all specialized to
+the case with two classes only. It is easy to see from the above that
+what we derived earlier is compatible with these equations.
+
+To find the optimal parameters we would typically use a gradient
+descent method.  Newton's method and gradient descent methods are
+discussed in the material on "optimization
+methods":"https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html".
+
+
+
+
+