lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update of lecture notes

Browse Source
This commit is contained in:
mhjensen
2020-12-20 23:07:22 +01:00
parent cb4dcd41ba
commit c001c2964f
23 changed files with 973 additions and 1406 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/LectureNotes/_build/.doctrees/chapter2.doctree
BIN
View File
Binary file not shown.
doc/src/LectureNotes/_build/.doctrees/environment.pickle
BIN
View File
Binary file not shown.
doc/src/LectureNotes/_build/html/_images/chapter2_178_0.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 5.0 KiB

After

Width:  |  Height:  |  Size: 5.1 KiB

doc/src/LectureNotes/_build/html/_images/chapter2_184_1.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 9.6 KiB

After

Width:  |  Height:  |  Size: 10 KiB

doc/src/LectureNotes/_build/html/_sources/chapter2.ipynb
+13 -13
View File
@@ -1669,7 +1669,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Random Numbers\n",
"## Random Numbers\n",
"\n",
"Uniform deviates are just random numbers that lie within a specified range\n",
"(typically 0 to 1), with any one number in the range just as likely as any other. They\n",
@@ -1686,7 +1686,7 @@
"\n",
"\n",
"\n",
"# Random Numbers, better name: pseudo random numbers\n",
"## Random Numbers, better name: pseudo random numbers\n",
"\n",
"A disclaimer is however appropriate. It should be fairly obvious that \n",
"something as deterministic as a computer cannot generate purely random numbers.\n",
@@ -1706,7 +1706,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG\n",
"## Random number generator RNG\n",
" The most common random number generators are based on so-called\n",
"Linear congruential relations of the type"
]
@@ -1750,7 +1750,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG and periodic outputs\n",
"## Random number generator RNG and periodic outputs\n",
"\n",
"The problem with such generators is that their outputs are periodic;\n",
"they \n",
@@ -1798,7 +1798,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG and its period\n",
"## Random number generator RNG and its period\n",
"Typical periods for the random generators provided in the program library \n",
"are of the order of $\\sim 10^9$ or larger. Other random number generators which have\n",
"become increasingly popular are so-called shift-register generators.\n",
@@ -1822,7 +1822,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Random number generator RNG, other examples\n",
"## Random number generator RNG, other examples\n",
"Such a generator again produces a sequence of pseudorandom numbers\n",
"but this time with a period much larger than $M$.\n",
"It is also possible to construct more elaborate algorithms by including\n",
@@ -1877,7 +1877,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, other examples\n",
"## Random number generator RNG, other examples\n",
"Instead of using modular addition, we could use the bitwise\n",
"exclusive-OR ($\\oplus$) operation so that"
]
@@ -1910,7 +1910,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, RAN0\n",
"## Random number generator RNG, RAN0\n",
"\n",
"We show here how the linear congruential algorithm can be implemented, namely"
]
@@ -1987,7 +1987,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, RAN0\n",
"## Random number generator RNG, RAN0\n",
"\n",
"To see how this works we note first that"
]
@@ -2019,7 +2019,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, RAN0\n",
"## Random number generator RNG, RAN0\n",
"We can now rewrite Eq. ([14](#eq:rntrick1)) as"
]
},
@@ -2087,7 +2087,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Random number generator RNG, RAN0\n",
"## Random number generator RNG, RAN0\n",
"The term $[N_{i-1}/q]r$ is always smaller or equal $N_{i-1}(r/q)$ and with $r < q$ we obtain always a \n",
"number smaller than $N_{i-1}$, which is smaller than $M$. \n",
"And since the number $N_{i-1}\\mathrm{MOD} (q)$ is between zero and $q-1$ then\n",
@@ -2105,7 +2105,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, RAN0 code"
"## Random number generator RNG, RAN0 code"
]
},
{
@@ -2457,7 +2457,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Which RNG should I use?\n",
"## Which RNG should I use?\n",
"* C++ has a class called **random**. The [random class](http://www.cplusplus.com/reference/random/) contains a large selection of RNGs and is highly recommended. Some of these RNGs have very large periods making it thereby very safe to use these RNGs in case one is performing large calculations. In particular, the [Mersenne twister random number engine](http://www.cplusplus.com/reference/random/mersenne_twister_engine/) has a period of $2^{19937}$. \n",
"\n",
"* Add RNGs in Python\n",
doc/src/LectureNotes/_build/html/chapter1.html
+6 -71
View File
@@ -89,99 +89,34 @@
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers">
2. Random Numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers-better-name-pseudo-random-numbers">
3. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng">
4. Random number generator RNG
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-periodic-outputs">
5. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-its-period">
6. Random number generator RNG and its period
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-other-examples">
7. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id9">
8. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0">
9. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id10">
10. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id11">
11. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id12">
12. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0-code">
13. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#which-rng-should-i-use">
14. Which RNG should I use?
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
15. Getting started, our first data and Machine Learning encounters
2. Getting started, our first data and Machine Learning encounters
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
16. Linear Regression and more Advanced Regression Analysis
3. Linear Regression and more Advanced Regression Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
17. Logistic Regression
4. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
18. Neural networks, from the simple perceptron to deep learning
5. Neural networks, from the simple perceptron to deep learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
19. Support Vector Machines, overarching aims
6. Support Vector Machines, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
20. Dimensionality Reduction
7. Dimensionality Reduction
</a>
</li>
</ul>
doc/src/LectureNotes/_build/html/chapter2.html
+255 -140
View File
@@ -38,7 +38,7 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="15. Getting started, our first data and Machine Learning encounters" href="chapter3.html" />
<link rel="next" title="2. Getting started, our first data and Machine Learning encounters" href="chapter3.html" />
<link rel="prev" title="Introduction to Applied Data Analysis and Machine Learning" href="chapter1.html" />
<meta name="viewport" content="width=device-width, initial-scale=1">
@@ -90,99 +90,34 @@
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#random-numbers">
2. Random Numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#random-numbers-better-name-pseudo-random-numbers">
3. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#random-number-generator-rng">
4. Random number generator RNG
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#random-number-generator-rng-and-periodic-outputs">
5. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#random-number-generator-rng-and-its-period">
6. Random number generator RNG and its period
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#random-number-generator-rng-other-examples">
7. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#id9">
8. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#random-number-generator-rng-ran0">
9. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#id10">
10. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#id11">
11. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#id12">
12. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#random-number-generator-rng-ran0-code">
13. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="#which-rng-should-i-use">
14. Which RNG should I use?
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
15. Getting started, our first data and Machine Learning encounters
2. Getting started, our first data and Machine Learning encounters
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
16. Linear Regression and more Advanced Regression Analysis
3. Linear Regression and more Advanced Regression Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
17. Logistic Regression
4. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
18. Neural networks, from the simple perceptron to deep learning
5. Neural networks, from the simple perceptron to deep learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
19. Support Vector Machines, overarching aims
6. Support Vector Machines, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
20. Dimensionality Reduction
7. Dimensionality Reduction
</a>
</li>
</ul>
@@ -455,6 +390,176 @@
1.40. Code to compute the Covariance matrix and the Covariance
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-numbers">
1.41. Random Numbers
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-numbers-better-name-pseudo-random-numbers">
1.42. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-number-generator-rng">
1.43. Random number generator RNG
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-number-generator-rng-and-periodic-outputs">
1.44. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-number-generator-rng-and-its-period">
1.45. Random number generator RNG and its period
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-number-generator-rng-other-examples">
1.46. Random number generator RNG, other examples
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id9">
1.47. Random number generator RNG, other examples
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-number-generator-rng-ran0">
1.48. Random number generator RNG, RAN0
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id10">
1.49. Random number generator RNG, RAN0
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id11">
1.50. Random number generator RNG, RAN0
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id12">
1.51. Random number generator RNG, RAN0
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#random-number-generator-rng-ran0-code">
1.52. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#properties-of-selected-random-number-generators">
1.53. Properties of Selected Random Number Generators
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id13">
1.54. Properties of Selected Random Number Generators
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id14">
1.55. Properties of Selected Random Number Generators
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-demonstration-of-rngs-using-python">
1.56. Simple demonstration of RNGs using python
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id15">
1.57. Properties of Selected Random Number Generators
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#autocorrelation-function">
1.58. Autocorrelation function
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-function-and-which-random-number-generators-should-i-use">
1.59. Correlation function and which random number generators should I use
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#which-rng-should-i-use">
1.60. Which RNG should I use?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#how-to-use-the-mersenne-generator">
1.61. How to use the Mersenne generator
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#why-blocking">
1.62. Why blocking?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id16">
1.63. Why blocking?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#code-to-demonstrate-the-calculation-of-the-autocorrelation-function">
1.64. Code to demonstrate the calculation of the autocorrelation function
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#what-is-blocking">
1.65. What is blocking?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id17">
1.66. What is blocking?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id18">
1.67. What is blocking?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#implementation">
1.68. Implementation
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#actual-implementation-with-code-main-function">
1.69. Actual implementation with code, main function
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-bootstrap-method">
1.70. The Bootstrap method
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bootstrapping">
1.71. Bootstrapping
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bootstrapping-recipe">
1.72. Bootstrapping, recipe
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bootstrapping-code">
1.73. Bootstrapping, code
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#jackknife-code">
1.74. Jackknife, code
</a>
</li>
</ul>
</nav>
@@ -1011,27 +1116,37 @@ uncorrelated.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>9.106279149747735
[[ 1.87336856 1.77787047 1.44771535 6.44982327 11.43684153
5.83694909 1.84228806 7.37557482 -2.58314202 6.48643762]
[ 1.77787047 1.68724056 1.37391564 6.12103277 10.85382947
5.53940088 1.74837435 6.9995926 -2.45146205 6.15578065]
[ 1.44771535 1.37391564 1.11877597 4.98434123 8.83824539
4.51071987 1.42369673 5.69975023 -1.99621924 5.01263633]
[ 6.44982327 6.12103277 4.98434123 22.2061057 39.37591798
20.09604032 6.34281617 25.39337703 -8.89350334 22.33216531]
[ 11.43684153 10.85382947 8.83824539 39.37591798 69.82146882
35.63434516 11.2470963 45.02759486 -15.76998069 39.59944717]
[ 5.83694909 5.53940088 4.51071987 20.09604032 35.63434516
18.18647726 5.74011 22.9804512 -8.04842615 20.21012151]
[ 1.84228806 1.74837435 1.42369673 6.34281617 11.2470963
5.74011 1.8117232 7.25320885 -2.54028588 6.37882306]
[ 7.37557482 6.9995926 5.69975023 25.39337703 45.02759486
22.9804512 7.25320885 29.03812156 -10.16999948 25.53753014]
[ -2.58314202 -2.45146205 -1.99621924 -8.89350334 -15.76998069
-8.04842615 -2.54028588 -10.16999948 3.56183127 -8.94398998]
[ 6.48643762 6.15578065 5.01263633 22.33216531 39.59944717
20.21012151 6.37882306 25.53753014 -8.94398998 22.45894054]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>2.56416115970942
[[ 5.90838144e+00 6.98010863e+00 -1.93947492e-01 1.33851317e+00
8.51355225e+00 8.12403403e+00 4.82725395e+00 7.54223480e+00
9.38189254e+00 5.58473257e+00]
[ 6.98010863e+00 8.24623749e+00 -2.29127820e-01 1.58130740e+00
1.00578340e+01 9.59766065e+00 5.70287435e+00 8.91032829e+00
1.10836834e+01 6.59775278e+00]
[-1.93947492e-01 -2.29127820e-01 6.36648631e-03 -4.39377982e-02
-2.79464371e-01 -2.66678115e-01 -1.58458590e-01 -2.47580075e-01
-3.07968357e-01 -1.83323451e-01]
[ 1.33851317e+00 1.58130740e+00 -4.39377982e-02 3.03233215e-01
1.92870110e+00 1.84045777e+00 1.09358934e+00 1.70865417e+00
2.12541909e+00 1.26519220e+00]
[ 8.51355225e+00 1.00578340e+01 -2.79464371e-01 1.92870110e+00
1.22674158e+01 1.17061481e+01 6.95572537e+00 1.08678173e+01
1.35186316e+01 8.04719753e+00]
[ 8.12403403e+00 9.59766065e+00 -2.66678115e-01 1.84045777e+00
1.17061481e+01 1.11705599e+01 6.63748198e+00 1.03705850e+01
1.29001174e+01 7.67901631e+00]
[ 4.82725395e+00 5.70287435e+00 -1.58458590e-01 1.09358934e+00
6.95572537e+00 6.63748198e+00 3.94395333e+00 6.16214154e+00
7.66517501e+00 4.56282700e+00]
[ 7.54223480e+00 8.91032829e+00 -2.47580075e-01 1.70865417e+00
1.08678173e+01 1.03705850e+01 6.16214154e+00 9.62790003e+00
1.19762810e+01 7.12908684e+00]
[ 9.38189254e+00 1.10836834e+01 -3.07968357e-01 2.12541909e+00
1.35186316e+01 1.29001174e+01 7.66517501e+00 1.19762810e+01
1.48974653e+01 8.86797194e+00]
[ 5.58473257e+00 6.59775278e+00 -1.83323451e-01 1.26519220e+00
8.04719753e+00 7.67901631e+00 4.56282700e+00 7.12908684e+00
8.86797194e+00 5.27881252e+00]]
</pre></div>
</div>
</div>
@@ -1339,23 +1454,22 @@ more practically oriented methods like the blocking technique.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.06894012083823547
4.139781119573823
0.04321525078901231
1.0006918520539008 10.50426403458282 18.88724031810869
3.0717579594084814 3.4645963550106305 10.40024628605229
[[ 1.00069185 3.07175796 3.46459636]
[ 3.07175796 10.50426403 10.40024629]
[ 3.46459636 10.40024629 18.88724032]]
[26.72833966 0.07676079 3.58709575]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.06895120441963641
3.74382855355186
-0.2186383556244991
0.8186111112149481 8.041491608761895 5.824398367359107
2.4292738814902837 1.6663000899456695 4.915994880963659
[[0.81861111 2.42927388 1.66630009]
[2.42927388 8.04149161 4.91599488]
[1.66630009 4.91599488 5.82439837]]
[12.69915666 0.06805927 1.91728516]
</pre></div>
</div>
</div>
</div>
</div>
</div>
<div class="section" id="random-numbers">
<h1><span class="section-number">2. </span>Random Numbers<a class="headerlink" href="#random-numbers" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.41. </span>Random Numbers<a class="headerlink" href="#random-numbers" title="Permalink to this headline"></a></h2>
<p>Uniform deviates are just random numbers that lie within a specified range
(typically 0 to 1), with any one number in the range just as likely as any other. They
are, in other words, what you probably think random numbers are. However,
@@ -1368,7 +1482,7 @@ of this section, is an essential building block for any sort of stochastic model
or Monte Carlo computer work.</p>
</div>
<div class="section" id="random-numbers-better-name-pseudo-random-numbers">
<h1><span class="section-number">3. </span>Random Numbers, better name: pseudo random numbers<a class="headerlink" href="#random-numbers-better-name-pseudo-random-numbers" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.42. </span>Random Numbers, better name: pseudo random numbers<a class="headerlink" href="#random-numbers-better-name-pseudo-random-numbers" title="Permalink to this headline"></a></h2>
<p>A disclaimer is however appropriate. It should be fairly obvious that
something as deterministic as a computer cannot generate purely random numbers.</p>
<p>Numbers generated by any of the standard algorithms are in reality pseudo random
@@ -1381,7 +1495,7 @@ numbers, hopefully abiding to the following criteria:</p>
</ul>
</div>
<div class="section" id="random-number-generator-rng">
<h1><span class="section-number">4. </span>Random number generator RNG<a class="headerlink" href="#random-number-generator-rng" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.43. </span>Random number generator RNG<a class="headerlink" href="#random-number-generator-rng" title="Permalink to this headline"></a></h2>
<p>The most common random number generators are based on so-called
Linear congruential relations of the type</p>
<div class="math notranslate nohighlight">
@@ -1401,7 +1515,7 @@ that is if we were to evaluate <span class="math notranslate nohighlight">\((13)
of the division <span class="math notranslate nohighlight">\(13/9\)</span>, namely <span class="math notranslate nohighlight">\(4\)</span>.</p>
</div>
<div class="section" id="random-number-generator-rng-and-periodic-outputs">
<h1><span class="section-number">5. </span>Random number generator RNG and periodic outputs<a class="headerlink" href="#random-number-generator-rng-and-periodic-outputs" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.44. </span>Random number generator RNG and periodic outputs<a class="headerlink" href="#random-number-generator-rng-and-periodic-outputs" title="Permalink to this headline"></a></h2>
<p>The problem with such generators is that their outputs are periodic;
they
will start to repeat themselves with a period that is at most <span class="math notranslate nohighlight">\(M\)</span>. If however
@@ -1423,7 +1537,7 @@ N_i=(27N_{i-1}+11) \mathrm{MOD} (54),
just <span class="math notranslate nohighlight">\(2\)</span>.</p>
</div>
<div class="section" id="random-number-generator-rng-and-its-period">
<h1><span class="section-number">6. </span>Random number generator RNG and its period<a class="headerlink" href="#random-number-generator-rng-and-its-period" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.45. </span>Random number generator RNG and its period<a class="headerlink" href="#random-number-generator-rng-and-its-period" title="Permalink to this headline"></a></h2>
<p>Typical periods for the random generators provided in the program library
are of the order of <span class="math notranslate nohighlight">\(\sim 10^9\)</span> or larger. Other random number generators which have
become increasingly popular are so-called shift-register generators.
@@ -1438,7 +1552,7 @@ N_l=(aN_{l-i}+cN_{l-j})\mathrm{MOD}(M).
\]</div>
</div>
<div class="section" id="random-number-generator-rng-other-examples">
<h1><span class="section-number">7. </span>Random number generator RNG, other examples<a class="headerlink" href="#random-number-generator-rng-other-examples" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.46. </span>Random number generator RNG, other examples<a class="headerlink" href="#random-number-generator-rng-other-examples" title="Permalink to this headline"></a></h2>
<p>Such a generator again produces a sequence of pseudorandom numbers
but this time with a period much larger than <span class="math notranslate nohighlight">\(M\)</span>.
It is also possible to construct more elaborate algorithms by including
@@ -1467,7 +1581,7 @@ which consists of two congruential relations</p>
<p>which according to the authors has a period larger than <span class="math notranslate nohighlight">\(2^{94}\)</span>.</p>
</div>
<div class="section" id="id9">
<h1><span class="section-number">8. </span>Random number generator RNG, other examples<a class="headerlink" href="#id9" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.47. </span>Random number generator RNG, other examples<a class="headerlink" href="#id9" title="Permalink to this headline"></a></h2>
<p>Instead of using modular addition, we could use the bitwise
exclusive-OR (<span class="math notranslate nohighlight">\(\oplus\)</span>) operation so that</p>
<div class="math notranslate nohighlight">
@@ -1485,7 +1599,7 @@ function <span class="math notranslate nohighlight">\(\mathrm{IEOR}(m,n)\)</span
it is given by <span class="math notranslate nohighlight">\(m\wedge n\)</span>.</p>
</div>
<div class="section" id="random-number-generator-rng-ran0">
<h1><span class="section-number">9. </span>Random number generator RNG, RAN0<a class="headerlink" href="#random-number-generator-rng-ran0" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.48. </span>Random number generator RNG, RAN0<a class="headerlink" href="#random-number-generator-rng-ran0" title="Permalink to this headline"></a></h2>
<p>We show here how the linear congruential algorithm can be implemented, namely</p>
<div class="math notranslate nohighlight">
\[
@@ -1513,7 +1627,7 @@ r = M\hspace{0.1cm}\mathrm{MOD} \hspace{0.1cm}a.
<span class="math notranslate nohighlight">\(q\)</span> and <span class="math notranslate nohighlight">\(r\)</span> are chosen so that <span class="math notranslate nohighlight">\(r &lt; q\)</span>.</p>
</div>
<div class="section" id="id10">
<h1><span class="section-number">10. </span>Random number generator RNG, RAN0<a class="headerlink" href="#id10" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.49. </span>Random number generator RNG, RAN0<a class="headerlink" href="#id10" title="Permalink to this headline"></a></h2>
<p>To see how this works we note first that</p>
<!-- Equation labels as ordinary links -->
<div id="eq:rntrick1"></div>
@@ -1529,7 +1643,7 @@ The last term <span class="math notranslate nohighlight">\([N_{i-1}/q]M\mathrm{M
<span class="math notranslate nohighlight">\([N_{i-1}/q]\)</span> just yields a constant which is multiplied with <span class="math notranslate nohighlight">\(M\)</span>.</p>
</div>
<div class="section" id="id11">
<h1><span class="section-number">11. </span>Random number generator RNG, RAN0<a class="headerlink" href="#id11" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.50. </span>Random number generator RNG, RAN0<a class="headerlink" href="#id11" title="Permalink to this headline"></a></h2>
<p>We can now rewrite Eq. (<a class="reference external" href="#eq:rntrick1">14</a>) as</p>
<!-- Equation labels as ordinary links -->
<div id="eq:rntrick2"></div>
@@ -1563,7 +1677,7 @@ in</p>
\]</div>
</div>
<div class="section" id="id12">
<h1><span class="section-number">12. </span>Random number generator RNG, RAN0<a class="headerlink" href="#id12" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.51. </span>Random number generator RNG, RAN0<a class="headerlink" href="#id12" title="Permalink to this headline"></a></h2>
<p>The term <span class="math notranslate nohighlight">\([N_{i-1}/q]r\)</span> is always smaller or equal <span class="math notranslate nohighlight">\(N_{i-1}(r/q)\)</span> and with <span class="math notranslate nohighlight">\(r &lt; q\)</span> we obtain always a
number smaller than <span class="math notranslate nohighlight">\(N_{i-1}\)</span>, which is smaller than <span class="math notranslate nohighlight">\(M\)</span>.
And since the number <span class="math notranslate nohighlight">\(N_{i-1}\mathrm{MOD} (q)\)</span> is between zero and <span class="math notranslate nohighlight">\(q-1\)</span> then
@@ -1578,7 +1692,7 @@ algorithm is that is should never be called with the initial seed
set to <span class="math notranslate nohighlight">\(0\)</span>.</p>
</div>
<div class="section" id="random-number-generator-rng-ran0-code">
<h1><span class="section-number">13. </span>Random number generator RNG, RAN0 code<a class="headerlink" href="#random-number-generator-rng-ran0-code" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.52. </span>Random number generator RNG, RAN0 code<a class="headerlink" href="#random-number-generator-rng-ran0-code" title="Permalink to this headline"></a></h2>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> /*
** The function
** ran0()
@@ -1607,8 +1721,9 @@ set to <span class="math notranslate nohighlight">\(0\)</span>.</p>
} // End: function ran0()
</pre></div>
</div>
</div>
<div class="section" id="properties-of-selected-random-number-generators">
<h2><span class="section-number">13.1. </span>Properties of Selected Random Number Generators<a class="headerlink" href="#properties-of-selected-random-number-generators" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.53. </span>Properties of Selected Random Number Generators<a class="headerlink" href="#properties-of-selected-random-number-generators" title="Permalink to this headline"></a></h2>
<p>As mentioned previously, the underlying PDF for the generation of
random numbers is the uniform distribution, meaning that the
probability for finding a number <span class="math notranslate nohighlight">\(x\)</span> in the interval [0,1] is <span class="math notranslate nohighlight">\(p(x)=1\)</span>.</p>
@@ -1622,7 +1737,7 @@ deviations.</p>
<span class="math notranslate nohighlight">\(\mu=\langle x\rangle\)</span>.</p>
</div>
<div class="section" id="id13">
<h2><span class="section-number">13.2. </span>Properties of Selected Random Number Generators<a class="headerlink" href="#id13" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.54. </span>Properties of Selected Random Number Generators<a class="headerlink" href="#id13" title="Permalink to this headline"></a></h2>
<p>For the uniform distribution, the mean value <span class="math notranslate nohighlight">\(\mu\)</span> is then</p>
<div class="math notranslate nohighlight">
\[
@@ -1635,7 +1750,7 @@ deviations.</p>
\]</div>
</div>
<div class="section" id="id14">
<h2><span class="section-number">13.3. </span>Properties of Selected Random Number Generators<a class="headerlink" href="#id14" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.55. </span>Properties of Selected Random Number Generators<a class="headerlink" href="#id14" title="Permalink to this headline"></a></h2>
<p>The various random number generators produce results which agree rather well with
these limiting values.</p>
<table border="1">
@@ -1659,7 +1774,7 @@ these limiting values.</p>
</table>
</div>
<div class="section" id="simple-demonstration-of-rngs-using-python">
<h2><span class="section-number">13.4. </span>Simple demonstration of RNGs using python<a class="headerlink" href="#simple-demonstration-of-rngs-using-python" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.56. </span>Simple demonstration of RNGs using python<a class="headerlink" href="#simple-demonstration-of-rngs-using-python" title="Permalink to this headline"></a></h2>
<p>The following simple Python code plots the distribution of the produced random numbers using the linear congruential RNG employed by Python. The trend displayed in the previous table is seen rather clearly.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -1694,7 +1809,7 @@ these limiting values.</p>
</div>
</div>
<div class="section" id="id15">
<h2><span class="section-number">13.5. </span>Properties of Selected Random Number Generators<a class="headerlink" href="#id15" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.57. </span>Properties of Selected Random Number Generators<a class="headerlink" href="#id15" title="Permalink to this headline"></a></h2>
<p>Since our random numbers, which are typically generated via a linear congruential algorithm,
are never fully independent, we can then define
an important test which measures the degree of correlation, namely the so-called<br />
@@ -1717,7 +1832,7 @@ in the evaluation of other expectation values. If they are not independent, our
assumption for approximating <span class="math notranslate nohighlight">\(\sigma_N\)</span> is no longer valid.</p>
</div>
<div class="section" id="autocorrelation-function">
<h2><span class="section-number">13.6. </span>Autocorrelation function<a class="headerlink" href="#autocorrelation-function" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.58. </span>Autocorrelation function<a class="headerlink" href="#autocorrelation-function" title="Permalink to this headline"></a></h2>
<p>This program computes the autocorrelation function as discussed in the equation on the previous slide for random numbers generated with the normal distribution <span class="math notranslate nohighlight">\(N(0,1)\)</span>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -1754,7 +1869,7 @@ assumption for approximating <span class="math notranslate nohighlight">\(\sigma
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.027481252820017347 1.0170183171235068
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.052276839521249285 0.9675166677604984
</pre></div>
</div>
<img alt="_images/chapter2_184_1.png" src="_images/chapter2_184_1.png" />
@@ -1764,7 +1879,7 @@ assumption for approximating <span class="math notranslate nohighlight">\(\sigma
For the remaining values we notice that there are still non-zero values for the auto-correlation function.</p>
</div>
<div class="section" id="correlation-function-and-which-random-number-generators-should-i-use">
<h2><span class="section-number">13.7. </span>Correlation function and which random number generators should I use<a class="headerlink" href="#correlation-function-and-which-random-number-generators-should-i-use" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.59. </span>Correlation function and which random number generators should I use<a class="headerlink" href="#correlation-function-and-which-random-number-generators-should-i-use" title="Permalink to this headline"></a></h2>
<p>The program here computes the correlation function for one of the standard functions included with the c++ compiler.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> // This function computes the autocorrelation function for
// the standard c++ random number generator
@@ -1835,15 +1950,15 @@ For the remaining values we notice that there are still non-zero values for the
</pre></div>
</div>
</div>
</div>
<div class="section" id="which-rng-should-i-use">
<h1><span class="section-number">14. </span>Which RNG should I use?<a class="headerlink" href="#which-rng-should-i-use" title="Permalink to this headline"></a></h1>
<h2><span class="section-number">1.60. </span>Which RNG should I use?<a class="headerlink" href="#which-rng-should-i-use" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li><p>C++ has a class called <strong>random</strong>. The <a class="reference external" href="http://www.cplusplus.com/reference/random/">random class</a> contains a large selection of RNGs and is highly recommended. Some of these RNGs have very large periods making it thereby very safe to use these RNGs in case one is performing large calculations. In particular, the <a class="reference external" href="http://www.cplusplus.com/reference/random/mersenne_twister_engine/">Mersenne twister random number engine</a> has a period of <span class="math notranslate nohighlight">\(2^{19937}\)</span>.</p></li>
<li><p>Add RNGs in Python</p></li>
</ul>
</div>
<div class="section" id="how-to-use-the-mersenne-generator">
<h2><span class="section-number">14.1. </span>How to use the Mersenne generator<a class="headerlink" href="#how-to-use-the-mersenne-generator" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.61. </span>How to use the Mersenne generator<a class="headerlink" href="#how-to-use-the-mersenne-generator" title="Permalink to this headline"></a></h2>
<p>The following part of a c++ code (from project 4) sets up the uniform distribution for <span class="math notranslate nohighlight">\(x\in [0,1]\)</span>.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> /*
@@ -1862,7 +1977,7 @@ For the remaining values we notice that there are still non-zero values for the
</div>
</div>
<div class="section" id="why-blocking">
<h2><span class="section-number">14.2. </span>Why blocking?<a class="headerlink" href="#why-blocking" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.62. </span>Why blocking?<a class="headerlink" href="#why-blocking" title="Permalink to this headline"></a></h2>
<p><strong>Statistical analysis.</strong></p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>* Monte Carlo simulations can be treated as *computer experiments*
@@ -1874,7 +1989,7 @@ For the remaining values we notice that there are still non-zero values for the
<p>A very good article which explains blocking is H. Flyvbjerg and H. G. Petersen, <em>Error estimates on averages of correlated data</em>, <a class="reference external" href="http://scitation.aip.org/content/aip/journal/jcp/91/1/10.1063/1.457480">Journal of Chemical Physics 91, 461-466 (1989)</a>.</p>
</div>
<div class="section" id="id16">
<h2><span class="section-number">14.3. </span>Why blocking?<a class="headerlink" href="#id16" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.63. </span>Why blocking?<a class="headerlink" href="#id16" title="Permalink to this headline"></a></h2>
<p><strong>Statistical analysis.</strong></p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>* As in other experiments, Monte Carlo experiments have two classes of errors:
@@ -1890,7 +2005,7 @@ For the remaining values we notice that there are still non-zero values for the
</div>
</div>
<div class="section" id="code-to-demonstrate-the-calculation-of-the-autocorrelation-function">
<h2><span class="section-number">14.4. </span>Code to demonstrate the calculation of the autocorrelation function<a class="headerlink" href="#code-to-demonstrate-the-calculation-of-the-autocorrelation-function" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.64. </span>Code to demonstrate the calculation of the autocorrelation function<a class="headerlink" href="#code-to-demonstrate-the-calculation-of-the-autocorrelation-function" title="Permalink to this headline"></a></h2>
<p>The following code computes the autocorrelation function, the covariance and the standard deviation
for standard RNG.
The <a class="reference external" href="https://github.com/CompPhysics/ComputationalPhysics2/tree/gh-pages/doc/Programs/LecturePrograms/programs/Blocking/autocorrelation.cpp">following file</a> gives the code.</p>
@@ -1975,7 +2090,7 @@ The <a class="reference external" href="https://github.com/CompPhysics/Computati
</div>
</div>
<div class="section" id="what-is-blocking">
<h2><span class="section-number">14.5. </span>What is blocking?<a class="headerlink" href="#what-is-blocking" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.65. </span>What is blocking?<a class="headerlink" href="#what-is-blocking" title="Permalink to this headline"></a></h2>
<p><strong>Blocking.</strong></p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>* Say that we have a set of samples from a Monte Carlo experiment
@@ -1996,7 +2111,7 @@ The <a class="reference external" href="https://github.com/CompPhysics/Computati
<p>where <span class="math notranslate nohighlight">\(\tau\)</span> is the correlation time (the time between a sample and the next uncorrelated sample) and <span class="math notranslate nohighlight">\(\Delta t\)</span> is time between each sample</p>
</div>
<div class="section" id="id17">
<h2><span class="section-number">14.6. </span>What is blocking?<a class="headerlink" href="#id17" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.66. </span>What is blocking?<a class="headerlink" href="#id17" title="Permalink to this headline"></a></h2>
<p><strong>Blocking.</strong></p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>* If $\Delta t\gg\tau$ our first estimate of $\sigma$ still holds
@@ -2013,7 +2128,7 @@ The <a class="reference external" href="https://github.com/CompPhysics/Computati
</div>
</div>
<div class="section" id="id18">
<h2><span class="section-number">14.7. </span>What is blocking?<a class="headerlink" href="#id18" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.67. </span>What is blocking?<a class="headerlink" href="#id18" title="Permalink to this headline"></a></h2>
<p><strong>Blocking.</strong></p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>* Problem: We don&#39;t know $\tau$ or it is too expensive to compute
@@ -2026,7 +2141,7 @@ The <a class="reference external" href="https://github.com/CompPhysics/Computati
</div>
</div>
<div class="section" id="implementation">
<h2><span class="section-number">14.8. </span>Implementation<a class="headerlink" href="#implementation" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.68. </span>Implementation<a class="headerlink" href="#implementation" title="Permalink to this headline"></a></h2>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>* Do a Monte Carlo simulation, storing all samples to file
* Do the statistical analysis on this file, independently of your Monte Carlo program
@@ -2045,7 +2160,7 @@ The <a class="reference external" href="https://github.com/CompPhysics/Computati
</div>
</div>
<div class="section" id="actual-implementation-with-code-main-function">
<h2><span class="section-number">14.9. </span>Actual implementation with code, main function<a class="headerlink" href="#actual-implementation-with-code-main-function" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.69. </span>Actual implementation with code, main function<a class="headerlink" href="#actual-implementation-with-code-main-function" title="Permalink to this headline"></a></h2>
<p>When the file gets large, it can be useful to write your data in binary mode instead of ascii characters.
The <a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py">following python file</a> reads data from file with the output from every Monte Carlo cycle.</p>
<div class="cell docutils container">
@@ -2094,7 +2209,7 @@ The <a class="reference external" href="https://github.com/CompPhysics/MachineLe
</div>
</div>
<div class="section" id="the-bootstrap-method">
<h2><span class="section-number">14.10. </span>The Bootstrap method<a class="headerlink" href="#the-bootstrap-method" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.70. </span>The Bootstrap method<a class="headerlink" href="#the-bootstrap-method" title="Permalink to this headline"></a></h2>
<p>The Bootstrap resampling method is also very popular. It is very simple:</p>
<ol class="simple">
<li><p>Start with your sample of measurements and compute the sample variance and the mean values</p></li>
@@ -2106,7 +2221,7 @@ that it produces the correct standard deviation.</p>
<p>This method is very useful for small ensembles of data points.</p>
</div>
<div class="section" id="bootstrapping">
<h2><span class="section-number">14.11. </span>Bootstrapping<a class="headerlink" href="#bootstrapping" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.71. </span>Bootstrapping<a class="headerlink" href="#bootstrapping" title="Permalink to this headline"></a></h2>
<p>Given a set of <span class="math notranslate nohighlight">\(N\)</span> data, assume that we are interested in some
observable <span class="math notranslate nohighlight">\(\theta\)</span> which may be estimated from that set. This observable can also be for example the result of a fit based on all <span class="math notranslate nohighlight">\(N\)</span> raw data.
Let us call the value of the observable obtained from the original
@@ -2115,7 +2230,7 @@ other samples by choosing randomly <span class="math notranslate nohighlight">\(
This costs essentially nothing, since we just recycle the original data set for the building of new sets.</p>
</div>
<div class="section" id="bootstrapping-recipe">
<h2><span class="section-number">14.12. </span>Bootstrapping, recipe<a class="headerlink" href="#bootstrapping-recipe" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.72. </span>Bootstrapping, recipe<a class="headerlink" href="#bootstrapping-recipe" title="Permalink to this headline"></a></h2>
<p>Let us assume we have done this <span class="math notranslate nohighlight">\(K\)</span> times and thus have <span class="math notranslate nohighlight">\(K\)</span> sets of <span class="math notranslate nohighlight">\(N\)</span>
data values each.
Of course some values will enter more than once in the new sets. For each of these sets one computes the observable <span class="math notranslate nohighlight">\(\theta\)</span> resulting in values <span class="math notranslate nohighlight">\(\theta_k\)</span> with <span class="math notranslate nohighlight">\(k = 1,...,K\)</span>. Then one determines</p>
@@ -2134,7 +2249,7 @@ sigma^2_{\tilde{\theta}} = \frac{1}{K} \sum_{k=1}^K \left(\theta_k-\tilde{\theta
the true <span class="math notranslate nohighlight">\(\angle\theta\rangle\)</span>. As final result for the observable one quotes <span class="math notranslate nohighlight">\(\angle\theta\rangle = \tilde{\theta} \pm \sigma_{\tilde{\theta}}\)</span> .</p>
</div>
<div class="section" id="bootstrapping-code">
<h2><span class="section-number">14.13. </span>Bootstrapping, <a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py">code</a><a class="headerlink" href="#bootstrapping-code" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.73. </span>Bootstrapping, <a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py">code</a><a class="headerlink" href="#bootstrapping-code" title="Permalink to this headline"></a></h2>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> # Bootstrap
@timeFunction
def bootstrap(self, nBoots = 1000):
@@ -2148,7 +2263,7 @@ the true <span class="math notranslate nohighlight">\(\angle\theta\rangle\)</spa
</div>
</div>
<div class="section" id="jackknife-code">
<h2><span class="section-number">14.14. </span>Jackknife, <a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py">code</a><a class="headerlink" href="#jackknife-code" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">1.74. </span>Jackknife, <a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py">code</a><a class="headerlink" href="#jackknife-code" title="Permalink to this headline"></a></h2>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> # Jackknife
@timeFunction
def jackknife(self):
@@ -2192,7 +2307,7 @@ the true <span class="math notranslate nohighlight">\(\angle\theta\rangle\)</spa
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter1.html" title="previous page">Introduction to Applied Data Analysis and Machine Learning</a>
<a class='right-next' id="next-link" href="chapter3.html" title="next page"><span class="section-number">15. </span>Getting started, our first data and Machine Learning encounters</a>
<a class='right-next' id="next-link" href="chapter3.html" title="next page"><span class="section-number">2. </span>Getting started, our first data and Machine Learning encounters</a>
</div>
<footer class="footer mt-5 mt-md-0">
doc/src/LectureNotes/_build/html/chapter3.html
+54 -119
View File
@@ -6,7 +6,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>15. Getting started, our first data and Machine Learning encounters &#8212; Applied Machine Learning and Data Analysis</title>
<title>2. Getting started, our first data and Machine Learning encounters &#8212; Applied Machine Learning and Data Analysis</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.11.2/css/all.min.css" integrity="sha384-KA6wR/X5RY4zFAHpv/CnoG2UW1uogYfdnP67Uv7eULvTveboZJg0qUpmJZb5VqzN" crossorigin="anonymous">
<link href="_static/css/index.css" rel="stylesheet">
<link rel="stylesheet" href="_static/sphinx-book-theme.css" type="text/css" />
@@ -38,7 +38,7 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="16. Linear Regression and more Advanced Regression Analysis" href="chapter4.html" />
<link rel="next" title="3. Linear Regression and more Advanced Regression Analysis" href="chapter4.html" />
<link rel="prev" title="1. Elements of Probability Theory and Statistical Data Analysis" href="chapter2.html" />
<meta name="viewport" content="width=device-width, initial-scale=1">
@@ -90,99 +90,34 @@
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers">
2. Random Numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers-better-name-pseudo-random-numbers">
3. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng">
4. Random number generator RNG
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-periodic-outputs">
5. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-its-period">
6. Random number generator RNG and its period
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-other-examples">
7. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id9">
8. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0">
9. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id10">
10. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id11">
11. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id12">
12. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0-code">
13. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#which-rng-should-i-use">
14. Which RNG should I use?
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
15. Getting started, our first data and Machine Learning encounters
2. Getting started, our first data and Machine Learning encounters
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
16. Linear Regression and more Advanced Regression Analysis
3. Linear Regression and more Advanced Regression Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
17. Logistic Regression
4. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
18. Neural networks, from the simple perceptron to deep learning
5. Neural networks, from the simple perceptron to deep learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
19. Support Vector Machines, overarching aims
6. Support Vector Machines, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
20. Dimensionality Reduction
7. Dimensionality Reduction
</a>
</li>
</ul>
@@ -257,89 +192,89 @@
<ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#introduction">
15.1. Introduction
2.1. Introduction
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#what-is-machine-learning">
15.2. What is Machine Learning?
2.2. What is Machine Learning?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#types-of-machine-learning">
15.3. Types of Machine Learning
2.3. Types of Machine Learning
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#software-and-needed-installations">
15.4. Software and needed installations
2.4. Software and needed installations
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#python-installers">
15.5. Python installers
2.5. Python installers
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#useful-python-libraries">
15.6. Useful Python libraries
2.6. Useful Python libraries
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#installing-r-c-cython-or-julia">
15.7. Installing R, C++, cython or Julia
2.7. Installing R, C++, cython or Julia
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#installing-r-c-cython-numba-etc">
15.8. Installing R, C++, cython, Numba etc
2.8. Installing R, C++, cython, Numba etc
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#numpy-examples-and-important-matrix-and-vector-handling-packages">
15.9. Numpy examples and Important Matrix and vector handling packages
2.9. Numpy examples and Important Matrix and vector handling packages
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-matrix-features">
15.10. Basic Matrix Features
2.10. Basic Matrix Features
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#some-famous-matrices">
15.10.1. Some famous Matrices
2.10.1. Some famous Matrices
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#more-basic-matrix-features">
15.10.2. More Basic Matrix Features
2.10.2. More Basic Matrix Features
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#numpy-and-arrays">
15.11. Numpy and arrays
2.11. Numpy and arrays
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#matrices-in-python">
15.12. Matrices in Python
2.12. Matrices in Python
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#meet-the-pandas">
15.13. Meet the Pandas
2.13. Meet the Pandas
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#reading-data-and-fitting">
15.14. Reading Data and fitting
2.14. Reading Data and fitting
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-linear-regression-model-using-scikit-learn">
15.14.1. Simple linear regression model using
2.14.1. Simple linear regression model using
<strong>
scikit-learn
</strong>
@@ -347,29 +282,29 @@
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies">
15.14.2. To our real data: nuclear binding energies. Brief reminder on masses and binding energies
2.14.2. To our real data: nuclear binding energies. Brief reminder on masses and binding energies
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#organizing-our-data">
15.14.3. Organizing our data
2.14.3. Organizing our data
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#seeing-the-wood-for-the-trees">
15.14.4. Seeing the wood for the trees
2.14.4. Seeing the wood for the trees
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#and-what-about-using-neural-networks">
15.14.5. And what about using neural networks?
2.14.5. And what about using neural networks?
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-first-summary">
15.15. A first summary
2.15. A first summary
</a>
</li>
</ul>
@@ -383,9 +318,9 @@
<div>
<div class="section" id="getting-started-our-first-data-and-machine-learning-encounters">
<h1><span class="section-number">15. </span>Getting started, our first data and Machine Learning encounters<a class="headerlink" href="#getting-started-our-first-data-and-machine-learning-encounters" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">2. </span>Getting started, our first data and Machine Learning encounters<a class="headerlink" href="#getting-started-our-first-data-and-machine-learning-encounters" title="Permalink to this headline"></a></h1>
<div class="section" id="introduction">
<h2><span class="section-number">15.1. </span>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.1. </span>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline"></a></h2>
<p>Our emphasis throughout this series of lectures<br />
is on understanding the mathematical aspects of
different algorithms used in the fields of data analysis and machine learning.</p>
@@ -417,7 +352,7 @@ well as allowing you to set up models and produce your own data and
get started with programming.</p>
</div>
<div class="section" id="what-is-machine-learning">
<h2><span class="section-number">15.2. </span>What is Machine Learning?<a class="headerlink" href="#what-is-machine-learning" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.2. </span>What is Machine Learning?<a class="headerlink" href="#what-is-machine-learning" title="Permalink to this headline"></a></h2>
<p>Statistics, data science and machine learning form important fields of
research in modern science. They describe how to learn and make
predictions from data, as well as allowing us to extract important
@@ -478,7 +413,7 @@ Carlo methods are central elements in a proper understanding of many
of algorithms and methods we will discuss.</p>
</div>
<div class="section" id="types-of-machine-learning">
<h2><span class="section-number">15.3. </span>Types of Machine Learning<a class="headerlink" href="#types-of-machine-learning" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.3. </span>Types of Machine Learning<a class="headerlink" href="#types-of-machine-learning" title="Permalink to this headline"></a></h2>
<p>The approaches to machine learning are many, but are often split into
two main categories. In <em>supervised learning</em> we know the answer to a
problem, and let the computer deduce the logic behind it. On the other
@@ -506,7 +441,7 @@ is in reproducing the data it is supposed to train.<br />
At the heart of basically all ML algorithms there are so-called minimization algorithms, often we end up with various variants of <strong>gradient</strong> methods.</p>
</div>
<div class="section" id="software-and-needed-installations">
<h2><span class="section-number">15.4. </span>Software and needed installations<a class="headerlink" href="#software-and-needed-installations" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.4. </span>Software and needed installations<a class="headerlink" href="#software-and-needed-installations" title="Permalink to this headline"></a></h2>
<p>We will make extensive use of Python as programming language and its
myriad of available libraries. You will find
Jupyter notebooks invaluable in your work. You can run <strong>R</strong>
@@ -535,7 +470,7 @@ you can use <strong>pip</strong> as well and simply install Python as</p>
<p>etc etc.</p>
</div>
<div class="section" id="python-installers">
<h2><span class="section-number">15.5. </span>Python installers<a class="headerlink" href="#python-installers" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.5. </span>Python installers<a class="headerlink" href="#python-installers" title="Permalink to this headline"></a></h2>
<p>If you don’t want to perform these operations separately and venture
into the hassle of exploring how to set up dependencies and paths, we
recommend two widely used distrubutions which set up all relevant
@@ -559,7 +494,7 @@ license.</p>
no setup and runs entirely in the cloud. Try it out!</p>
</div>
<div class="section" id="useful-python-libraries">
<h2><span class="section-number">15.6. </span>Useful Python libraries<a class="headerlink" href="#useful-python-libraries" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.6. </span>Useful Python libraries<a class="headerlink" href="#useful-python-libraries" title="Permalink to this headline"></a></h2>
<p>Here we list several useful Python libraries we strongly recommend (if you use anaconda many of these are already there)</p>
<ul class="simple">
<li><p><a class="reference external" href="https://www.numpy.org/">NumPy</a> is a highly popular library for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays</p></li>
@@ -576,7 +511,7 @@ no setup and runs entirely in the cloud. Try it out!</p>
</ul>
</div>
<div class="section" id="installing-r-c-cython-or-julia">
<h2><span class="section-number">15.7. </span>Installing R, C++, cython or Julia<a class="headerlink" href="#installing-r-c-cython-or-julia" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.7. </span>Installing R, C++, cython or Julia<a class="headerlink" href="#installing-r-c-cython-or-julia" title="Permalink to this headline"></a></h2>
<p>You will also find it convenient to utilize <strong>R</strong>. We will mainly
use Python during our lectures and in various projects and exercises.
Those of you
@@ -591,7 +526,7 @@ lectures.</p>
<a class="reference external" href="https://mpacer.org/maths/r-kernel-for-ipython-notebook">follow the link here</a></p>
</div>
<div class="section" id="installing-r-c-cython-numba-etc">
<h2><span class="section-number">15.8. </span>Installing R, C++, cython, Numba etc<a class="headerlink" href="#installing-r-c-cython-numba-etc" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.8. </span>Installing R, C++, cython, Numba etc<a class="headerlink" href="#installing-r-c-cython-numba-etc" title="Permalink to this headline"></a></h2>
<p>For the C++ aficionados, Jupyter/IPython notebook allows you also to
install C++ and run codes written in this language interactively in
the browser. Since we will emphasize writing many of the algorithms
@@ -616,7 +551,7 @@ further processing. For example, convert to latex as</p>
formats, ipython notebooks, latex files, pdf files etc with minimal edits. These lectures were generated using <strong>doconce</strong>.</p>
</div>
<div class="section" id="numpy-examples-and-important-matrix-and-vector-handling-packages">
<h2><span class="section-number">15.9. </span>Numpy examples and Important Matrix and vector handling packages<a class="headerlink" href="#numpy-examples-and-important-matrix-and-vector-handling-packages" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.9. </span>Numpy examples and Important Matrix and vector handling packages<a class="headerlink" href="#numpy-examples-and-important-matrix-and-vector-handling-packages" title="Permalink to this headline"></a></h2>
<p>There are several central software libraries for linear algebra and eigenvalue problems. Several of the more
popular ones have been wrapped into ofter software packages like those from the widely used text <strong>Numerical Recipes</strong>. The original source codes in many of the available packages are often taken from the widely used
software package LAPACK, which follows two other popular packages
@@ -628,7 +563,7 @@ developed in the 1970s, namely EISPACK and LINPACK. We describe them shortly he
</ul>
</div>
<div class="section" id="basic-matrix-features">
<h2><span class="section-number">15.10. </span>Basic Matrix Features<a class="headerlink" href="#basic-matrix-features" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.10. </span>Basic Matrix Features<a class="headerlink" href="#basic-matrix-features" title="Permalink to this headline"></a></h2>
<p><strong>Matrix properties reminder.</strong></p>
<div class="math notranslate nohighlight">
\[\begin{split}
@@ -663,7 +598,7 @@ developed in the 1970s, namely EISPACK and LINPACK. We describe them shortly he
</tbody>
</table>
<div class="section" id="some-famous-matrices">
<h3><span class="section-number">15.10.1. </span>Some famous Matrices<a class="headerlink" href="#some-famous-matrices" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">2.10.1. </span>Some famous Matrices<a class="headerlink" href="#some-famous-matrices" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li><p>Diagonal if <span class="math notranslate nohighlight">\(a_{ij}=0\)</span> for <span class="math notranslate nohighlight">\(i\ne j\)</span></p></li>
<li><p>Upper triangular if <span class="math notranslate nohighlight">\(a_{ij}=0\)</span> for <span class="math notranslate nohighlight">\(i &gt; j\)</span></p></li>
@@ -677,7 +612,7 @@ developed in the 1970s, namely EISPACK and LINPACK. We describe them shortly he
</ul>
</div>
<div class="section" id="more-basic-matrix-features">
<h3><span class="section-number">15.10.2. </span>More Basic Matrix Features<a class="headerlink" href="#more-basic-matrix-features" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">2.10.2. </span>More Basic Matrix Features<a class="headerlink" href="#more-basic-matrix-features" title="Permalink to this headline"></a></h3>
<p><strong>Some Equivalent Statements.</strong></p>
<p>For an <span class="math notranslate nohighlight">\(N\times N\)</span> matrix <span class="math notranslate nohighlight">\(\mathbf{A}\)</span> the following properties are all equivalent</p>
<ul class="simple">
@@ -691,7 +626,7 @@ developed in the 1970s, namely EISPACK and LINPACK. We describe them shortly he
</div>
</div>
<div class="section" id="numpy-and-arrays">
<h2><span class="section-number">15.11. </span>Numpy and arrays<a class="headerlink" href="#numpy-and-arrays" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.11. </span>Numpy and arrays<a class="headerlink" href="#numpy-and-arrays" title="Permalink to this headline"></a></h2>
<p><a class="reference external" href="http://www.numpy.org/">Numpy</a> provides an easy way to handle arrays in Python. The standard way to import this library is as</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -818,7 +753,7 @@ The attentive reader will also notice that the output is <span class="math notra
</div>
</div>
<div class="section" id="matrices-in-python">
<h2><span class="section-number">15.12. </span>Matrices in Python<a class="headerlink" href="#matrices-in-python" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.12. </span>Matrices in Python<a class="headerlink" href="#matrices-in-python" title="Permalink to this headline"></a></h2>
<p>Having defined vectors, we are now ready to try out matrices. We can
define a <span class="math notranslate nohighlight">\(3 \times 3 \)</span> real matrix <span class="math notranslate nohighlight">\(\hat{A}\)</span> as (recall that we user
lowercase letters for vectors and uppercase letters for matrices)</p>
@@ -963,7 +898,7 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
</div>
</div>
<div class="section" id="meet-the-pandas">
<h2><span class="section-number">15.13. </span>Meet the Pandas<a class="headerlink" href="#meet-the-pandas" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.13. </span>Meet the Pandas<a class="headerlink" href="#meet-the-pandas" title="Permalink to this headline"></a></h2>
<!-- dom:FIGURE: [fig/pandas.jpg, width=600 frac=0.8] -->
<!-- begin figure -->
<p></p>
@@ -1091,7 +1026,7 @@ As we will see below it leads also to a very concice code close to the mathemati
For multidimensional arrays, we recommend strongly <a class="reference external" href="http://xarray.pydata.org/en/stable/">xarray</a>. <strong>xarray</strong> has much of the same flexibility as <strong>pandas</strong>, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both <strong>pandas</strong> and <strong>xarray</strong>.</p>
</div>
<div class="section" id="reading-data-and-fitting">
<h2><span class="section-number">15.14. </span>Reading Data and fitting<a class="headerlink" href="#reading-data-and-fitting" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.14. </span>Reading Data and fitting<a class="headerlink" href="#reading-data-and-fitting" title="Permalink to this headline"></a></h2>
<p>In order to study various Machine Learning algorithms, we need to
access data. Acccessing data is an essential step in all machine
learning algorithms. In particular, setting up the so-called <strong>design
@@ -1117,7 +1052,7 @@ you can easily implement other algorithms like decision trees and random forests
<p>But before we really start with nuclear physics data, let’s just look at some simpler polynomial fitting cases, such as,
(don’t be offended) fitting straight lines!</p>
<div class="section" id="simple-linear-regression-model-using-scikit-learn">
<h3><span class="section-number">15.14.1. </span>Simple linear regression model using <strong>scikit-learn</strong><a class="headerlink" href="#simple-linear-regression-model-using-scikit-learn" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">2.14.1. </span>Simple linear regression model using <strong>scikit-learn</strong><a class="headerlink" href="#simple-linear-regression-model-using-scikit-learn" title="Permalink to this headline"></a></h3>
<p>We start with perhaps our simplest possible example, using <strong>Scikit-Learn</strong> to perform linear regression analysis on a data set produced by us.</p>
<p>What follows is a simple Python code where we have defined a function
<span class="math notranslate nohighlight">\(y\)</span> in terms of the variable <span class="math notranslate nohighlight">\(x\)</span>. Both are defined as vectors with <span class="math notranslate nohighlight">\(100\)</span> entries.
@@ -1400,7 +1335,7 @@ a linear <span class="math notranslate nohighlight">\(x\)</span>-dependence we s
</div>
</div>
<div class="section" id="to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies">
<h3><span class="section-number">15.14.2. </span>To our real data: nuclear binding energies. Brief reminder on masses and binding energies<a class="headerlink" href="#to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">2.14.2. </span>To our real data: nuclear binding energies. Brief reminder on masses and binding energies<a class="headerlink" href="#to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies" title="Permalink to this headline"></a></h3>
<p>Let us now dive into nuclear physics and remind ourselves briefly about some basic features about binding
energies. A basic quantity which can be measured for the ground
states of nuclei is the atomic mass <span class="math notranslate nohighlight">\(M(N, Z)\)</span> of the neutral atom with
@@ -1463,7 +1398,7 @@ arises from the tendency of proton pairs and neutron pairs to
occur. An even number of particles is more stable than an odd number.</p>
</div>
<div class="section" id="organizing-our-data">
<h3><span class="section-number">15.14.3. </span>Organizing our data<a class="headerlink" href="#organizing-our-data" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">2.14.3. </span>Organizing our data<a class="headerlink" href="#organizing-our-data" title="Permalink to this headline"></a></h3>
<p>Let us start with reading and organizing our data.
We start with the compilation of masses and binding energies from 2016.
After having downloaded this file to our own computer, we are now ready to read the file and start structuring our data.</p>
@@ -1650,7 +1585,7 @@ Now we can print measures of how our fit is doing, the coefficients from the fit
</div>
</div>
<div class="section" id="seeing-the-wood-for-the-trees">
<h3><span class="section-number">15.14.4. </span>Seeing the wood for the trees<a class="headerlink" href="#seeing-the-wood-for-the-trees" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">2.14.4. </span>Seeing the wood for the trees<a class="headerlink" href="#seeing-the-wood-for-the-trees" title="Permalink to this headline"></a></h3>
<p>As a teaser, let us now see how we can do this with decision trees using <strong>scikit-learn</strong>. Later we will switch to so-called <strong>random forests</strong>!</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -1689,7 +1624,7 @@ Now we can print measures of how our fit is doing, the coefficients from the fit
</div>
</div>
<div class="section" id="and-what-about-using-neural-networks">
<h3><span class="section-number">15.14.5. </span>And what about using neural networks?<a class="headerlink" href="#and-what-about-using-neural-networks" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">2.14.5. </span>And what about using neural networks?<a class="headerlink" href="#and-what-about-using-neural-networks" title="Permalink to this headline"></a></h3>
<p>The <strong>seaborn</strong> package allows us to visualize data in an efficient way. Note that we use <strong>scikit-learn</strong>’s multi-layer perceptron (or feed forward neural network)
functionality.</p>
<div class="cell docutils container">
@@ -1730,7 +1665,7 @@ functionality.</p>
</div>
</div>
<div class="section" id="a-first-summary">
<h2><span class="section-number">15.15. </span>A first summary<a class="headerlink" href="#a-first-summary" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">2.15. </span>A first summary<a class="headerlink" href="#a-first-summary" title="Permalink to this headline"></a></h2>
<p>The aim behind these introductory words was to present to you various
Python libraries and their functionalities, in particular libraries like
<strong>numpy</strong>, <strong>pandas</strong>, <strong>xarray</strong> and <strong>matplotlib</strong> and other that make our life much easier
@@ -1771,7 +1706,7 @@ Now it is time to dive more into the details of various methods. We will start w
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter2.html" title="previous page"><span class="section-number">1. </span>Elements of Probability Theory and Statistical Data Analysis</a>
<a class='right-next' id="next-link" href="chapter4.html" title="next page"><span class="section-number">16. </span>Linear Regression and more Advanced Regression Analysis</a>
<a class='right-next' id="next-link" href="chapter4.html" title="next page"><span class="section-number">3. </span>Linear Regression and more Advanced Regression Analysis</a>
</div>
<footer class="footer mt-5 mt-md-0">
doc/src/LectureNotes/_build/html/chapter4.html
+202 -267
View File
File diff suppressed because it is too large Load Diff
doc/src/LectureNotes/_build/html/chapter5.html
+24 -89
View File
@@ -6,7 +6,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>17. Logistic Regression &#8212; Applied Machine Learning and Data Analysis</title>
<title>4. Logistic Regression &#8212; Applied Machine Learning and Data Analysis</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.11.2/css/all.min.css" integrity="sha384-KA6wR/X5RY4zFAHpv/CnoG2UW1uogYfdnP67Uv7eULvTveboZJg0qUpmJZb5VqzN" crossorigin="anonymous">
<link href="_static/css/index.css" rel="stylesheet">
<link rel="stylesheet" href="_static/sphinx-book-theme.css" type="text/css" />
@@ -38,8 +38,8 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="18. Neural networks, from the simple perceptron to deep learning" href="chapter6.html" />
<link rel="prev" title="16. Linear Regression and more Advanced Regression Analysis" href="chapter4.html" />
<link rel="next" title="5. Neural networks, from the simple perceptron to deep learning" href="chapter6.html" />
<link rel="prev" title="3. Linear Regression and more Advanced Regression Analysis" href="chapter4.html" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="docsearch:language" content="en">
@@ -90,99 +90,34 @@
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers">
2. Random Numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers-better-name-pseudo-random-numbers">
3. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng">
4. Random number generator RNG
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-periodic-outputs">
5. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-its-period">
6. Random number generator RNG and its period
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-other-examples">
7. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id9">
8. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0">
9. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id10">
10. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id11">
11. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id12">
12. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0-code">
13. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#which-rng-should-i-use">
14. Which RNG should I use?
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
15. Getting started, our first data and Machine Learning encounters
2. Getting started, our first data and Machine Learning encounters
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
16. Linear Regression and more Advanced Regression Analysis
3. Linear Regression and more Advanced Regression Analysis
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
17. Logistic Regression
4. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
18. Neural networks, from the simple perceptron to deep learning
5. Neural networks, from the simple perceptron to deep learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
19. Support Vector Machines, overarching aims
6. Support Vector Machines, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
20. Dimensionality Reduction
7. Dimensionality Reduction
</a>
</li>
</ul>
@@ -257,32 +192,32 @@
<ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#introduction">
17.1. Introduction
4.1. Introduction
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basics">
17.2. Basics
4.2. Basics
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-logistic-function">
17.3. The logistic function
4.3. The logistic function
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#two-parameters">
17.4. Two parameters
4.4. Two parameters
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#maximum-likelihood">
17.5. Maximum likelihood
4.5. Maximum likelihood
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#including-more-classes">
17.6. Including more classes
4.6. Including more classes
</a>
</li>
</ul>
@@ -296,9 +231,9 @@
<div>
<div class="section" id="logistic-regression">
<h1><span class="section-number">17. </span>Logistic Regression<a class="headerlink" href="#logistic-regression" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">4. </span>Logistic Regression<a class="headerlink" href="#logistic-regression" title="Permalink to this headline"></a></h1>
<div class="section" id="introduction">
<h2><span class="section-number">17.1. </span>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">4.1. </span>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline"></a></h2>
<p>In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
@@ -338,7 +273,7 @@ regression are also commonly used in modern supervised Deep Learning
models, as we will see later.</p>
</div>
<div class="section" id="basics">
<h2><span class="section-number">17.2. </span>Basics<a class="headerlink" href="#basics" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">4.2. </span>Basics<a class="headerlink" href="#basics" title="Permalink to this headline"></a></h2>
<p>We consider the case where the dependent variables, also called the
responses or the outcomes, <span class="math notranslate nohighlight">\(y_i\)</span> are discrete and only take values
from <span class="math notranslate nohighlight">\(k=0,\dots,K-1\)</span> (i.e. <span class="math notranslate nohighlight">\(K\)</span> classes).</p>
@@ -387,7 +322,7 @@ favorable to use a ``soft” classifier that outputs
the probability of a given category. This leads us to the logistic function.</p>
</div>
<div class="section" id="the-logistic-function">
<h2><span class="section-number">17.3. </span>The logistic function<a class="headerlink" href="#the-logistic-function" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">4.3. </span>The logistic function<a class="headerlink" href="#the-logistic-function" title="Permalink to this headline"></a></h2>
<p>The perceptron is an example of a ``hard classification” model. We
will encounter this model when we discuss neural networks as
well. Each datapoint is deterministically assigned to a category (i.e
@@ -473,7 +408,7 @@ p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
</div>
</div>
<div class="section" id="two-parameters">
<h2><span class="section-number">17.4. </span>Two parameters<a class="headerlink" href="#two-parameters" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">4.4. </span>Two parameters<a class="headerlink" href="#two-parameters" title="Permalink to this headline"></a></h2>
<p>We assume now that we have two classes with <span class="math notranslate nohighlight">\(y_i\)</span> either <span class="math notranslate nohighlight">\(0\)</span> or <span class="math notranslate nohighlight">\(1\)</span>. Furthermore we assume also that we have only two parameters <span class="math notranslate nohighlight">\(\beta\)</span> in our fitting of the Sigmoid function, that is we define probabilities</p>
<div class="math notranslate nohighlight">
\[\begin{split}
@@ -490,7 +425,7 @@ p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
\]</div>
</div>
<div class="section" id="maximum-likelihood">
<h2><span class="section-number">17.5. </span>Maximum likelihood<a class="headerlink" href="#maximum-likelihood" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">4.5. </span>Maximum likelihood<a class="headerlink" href="#maximum-likelihood" title="Permalink to this headline"></a></h2>
<p>In order to define the total likelihood for all possible outcomes from a<br />
dataset <span class="math notranslate nohighlight">\(\mathcal{D}=\{(y_i,x_i)\}\)</span>, with the binary labels
<span class="math notranslate nohighlight">\(y_i\in\{0,1\}\)</span> and where the data points are drawn independently, we use the so-called <a class="reference external" href="https://en.wikipedia.org/wiki/Maximum_likelihood_estimation">Maximum Likelihood Estimation</a> (MLE) principle.
@@ -560,7 +495,7 @@ p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_
\]</div>
</div>
<div class="section" id="including-more-classes">
<h2><span class="section-number">17.6. </span>Including more classes<a class="headerlink" href="#including-more-classes" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">4.6. </span>Including more classes<a class="headerlink" href="#including-more-classes" title="Permalink to this headline"></a></h2>
<p>Till now we have mainly focused on two classes, the so-called binary
system. Suppose we wish to extend to <span class="math notranslate nohighlight">\(K\)</span> classes. Let us for the sake
of simplicity assume we have only two predictors. We have then
@@ -651,8 +586,8 @@ methods</a>.</p>
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter4.html" title="previous page"><span class="section-number">16. </span>Linear Regression and more Advanced Regression Analysis</a>
<a class='right-next' id="next-link" href="chapter6.html" title="next page"><span class="section-number">18. </span>Neural networks, from the simple perceptron to deep learning</a>
<a class='left-prev' id="prev-link" href="chapter4.html" title="previous page"><span class="section-number">3. </span>Linear Regression and more Advanced Regression Analysis</a>
<a class='right-next' id="next-link" href="chapter6.html" title="next page"><span class="section-number">5. </span>Neural networks, from the simple perceptron to deep learning</a>
</div>
<footer class="footer mt-5 mt-md-0">
doc/src/LectureNotes/_build/html/chapter6.html
+146 -211
View File
File diff suppressed because it is too large Load Diff
doc/src/LectureNotes/_build/html/chapter7.html
+64 -129
View File
@@ -6,7 +6,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>19. Support Vector Machines, overarching aims &#8212; Applied Machine Learning and Data Analysis</title>
<title>6. Support Vector Machines, overarching aims &#8212; Applied Machine Learning and Data Analysis</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.11.2/css/all.min.css" integrity="sha384-KA6wR/X5RY4zFAHpv/CnoG2UW1uogYfdnP67Uv7eULvTveboZJg0qUpmJZb5VqzN" crossorigin="anonymous">
<link href="_static/css/index.css" rel="stylesheet">
<link rel="stylesheet" href="_static/sphinx-book-theme.css" type="text/css" />
@@ -38,8 +38,8 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="20. Dimensionality Reduction" href="chapter8.html" />
<link rel="prev" title="18. Neural networks, from the simple perceptron to deep learning" href="chapter6.html" />
<link rel="next" title="7. Dimensionality Reduction" href="chapter8.html" />
<link rel="prev" title="5. Neural networks, from the simple perceptron to deep learning" href="chapter6.html" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="docsearch:language" content="en">
@@ -90,99 +90,34 @@
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers">
2. Random Numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers-better-name-pseudo-random-numbers">
3. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng">
4. Random number generator RNG
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-periodic-outputs">
5. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-its-period">
6. Random number generator RNG and its period
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-other-examples">
7. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id9">
8. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0">
9. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id10">
10. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id11">
11. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id12">
12. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0-code">
13. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#which-rng-should-i-use">
14. Which RNG should I use?
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
15. Getting started, our first data and Machine Learning encounters
2. Getting started, our first data and Machine Learning encounters
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
16. Linear Regression and more Advanced Regression Analysis
3. Linear Regression and more Advanced Regression Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
17. Logistic Regression
4. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
18. Neural networks, from the simple perceptron to deep learning
5. Neural networks, from the simple perceptron to deep learning
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
19. Support Vector Machines, overarching aims
6. Support Vector Machines, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
20. Dimensionality Reduction
7. Dimensionality Reduction
</a>
</li>
</ul>
@@ -257,17 +192,17 @@
<ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#hyperplanes-and-all-that">
19.1. Hyperplanes and all that
6.1. Hyperplanes and all that
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#what-is-a-hyperplane">
19.2. What is a hyperplane?
6.2. What is a hyperplane?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-p-dimensional-space-of-features">
19.3. A
6.3. A
<span class="math notranslate nohighlight">
\(p\)
</span>
@@ -276,117 +211,117 @@
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-two-dimensional-case">
19.4. The two-dimensional case
6.4. The two-dimensional case
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#getting-into-the-details">
19.5. Getting into the details
6.5. Getting into the details
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#first-attempt-at-a-minimization-approach">
19.6. First attempt at a minimization approach
6.6. First attempt at a minimization approach
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#solving-the-equations">
19.7. Solving the equations
6.7. Solving the equations
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#code-example">
19.8. Code Example
6.8. Code Example
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#problems-with-the-simpler-approach">
19.9. Problems with the Simpler Approach
6.9. Problems with the Simpler Approach
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-better-approach">
19.10. A better approach
6.10. A better approach
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-quick-reminder-on-lagrangian-multipliers">
19.11. A quick Reminder on Lagrangian Multipliers
6.11. A quick Reminder on Lagrangian Multipliers
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adding-the-multiplier">
19.12. Adding the Multiplier
6.12. Adding the Multiplier
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#setting-up-the-problem">
19.13. Setting up the Problem
6.13. Setting up the Problem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-problem-to-solve">
19.14. The problem to solve
6.14. The problem to solve
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-last-steps">
19.15. The last steps
6.15. The last steps
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-soft-classifier">
19.16. A soft classifier
6.16. A soft classifier
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#soft-optmization-problem">
19.17. Soft optmization problem
6.17. Soft optmization problem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#kernels-and-non-linearity">
19.18. Kernels and non-linearity
6.18. Kernels and non-linearity
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-equations">
19.19. The equations
6.19. The equations
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id1">
19.20. The problem to solve
6.20. The problem to solve
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#different-kernels-and-mercer-s-theorem">
19.21. Different kernels and Mercer’s theorem
6.21. Different kernels and Mercer’s theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-moons-example">
19.22. The moons example
6.22. The moons example
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#mathematical-optimization-of-convex-functions">
19.23. Mathematical optimization of convex functions
6.23. Mathematical optimization of convex functions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#how-do-we-solve-these-problems">
19.24. How do we solve these problems?
6.24. How do we solve these problems?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-simple-example">
19.25. A simple example
6.25. A simple example
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#back-to-the-more-realistic-cases">
19.26. Back to the more realistic cases
6.26. Back to the more realistic cases
</a>
</li>
</ul>
@@ -400,7 +335,7 @@
<div>
<div class="section" id="support-vector-machines-overarching-aims">
<h1><span class="section-number">19. </span>Support Vector Machines, overarching aims<a class="headerlink" href="#support-vector-machines-overarching-aims" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">6. </span>Support Vector Machines, overarching aims<a class="headerlink" href="#support-vector-machines-overarching-aims" title="Permalink to this headline"></a></h1>
<p>A Support Vector Machine (SVM) is a very powerful and versatile
Machine Learning method, capable of performing linear or nonlinear
classification, regression, and even outlier detection. It is one of
@@ -422,7 +357,7 @@ below. We distinguish also between linear and non-linear
approaches. The latter are the most frequent ones since it is rather
unlikely that we can separate classes easily by say straight lines.</p>
<div class="section" id="hyperplanes-and-all-that">
<h2><span class="section-number">19.1. </span>Hyperplanes and all that<a class="headerlink" href="#hyperplanes-and-all-that" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.1. </span>Hyperplanes and all that<a class="headerlink" href="#hyperplanes-and-all-that" title="Permalink to this headline"></a></h2>
<p>The theory behind support vector machines (SVM hereafter) is based on
the mathematical description of so-called hyperplanes. Let us start
with a two-dimensional case. This will also allow us to introduce our
@@ -516,7 +451,7 @@ SGDClassifier(alpha=0.00200): [0.117] [[0.77714169 0.72981762]]
</div>
</div>
<div class="section" id="what-is-a-hyperplane">
<h2><span class="section-number">19.2. </span>What is a hyperplane?<a class="headerlink" href="#what-is-a-hyperplane" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.2. </span>What is a hyperplane?<a class="headerlink" href="#what-is-a-hyperplane" title="Permalink to this headline"></a></h2>
<p>The aim of the SVM algorithm is to find a hyperplane in a
<span class="math notranslate nohighlight">\(p\)</span>-dimensional space, where <span class="math notranslate nohighlight">\(p\)</span> is the number of features that
distinctly classifies the data points.</p>
@@ -538,7 +473,7 @@ We can then rewrite the above equation as</p>
\]</div>
</div>
<div class="section" id="a-p-dimensional-space-of-features">
<h2><span class="section-number">19.3. </span>A <span class="math notranslate nohighlight">\(p\)</span>-dimensional space of features<a class="headerlink" href="#a-p-dimensional-space-of-features" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.3. </span>A <span class="math notranslate nohighlight">\(p\)</span>-dimensional space of features<a class="headerlink" href="#a-p-dimensional-space-of-features" title="Permalink to this headline"></a></h2>
<p>We limit ourselves to two classes of outputs <span class="math notranslate nohighlight">\(y_i\)</span> and assign these classes the values <span class="math notranslate nohighlight">\(y_i = \pm 1\)</span>.
In a <span class="math notranslate nohighlight">\(p\)</span>-dimensional space of say <span class="math notranslate nohighlight">\(p\)</span> features we have a hyperplane defines as</p>
<div class="math notranslate nohighlight">
@@ -573,7 +508,7 @@ y_i\left(b+w_1x_{i1}+w_2x_{i2}+\dots +w_px_{ip}\right) &gt; 0.
<p>When we try to separate hyperplanes, if it exists, we can use it to construct a natural classifier: a test observation is assigned a given class depending on which side of the hyperplane it is located.</p>
</div>
<div class="section" id="the-two-dimensional-case">
<h2><span class="section-number">19.4. </span>The two-dimensional case<a class="headerlink" href="#the-two-dimensional-case" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.4. </span>The two-dimensional case<a class="headerlink" href="#the-two-dimensional-case" title="Permalink to this headline"></a></h2>
<p>Let us try to develop our intuition about SVMs by limiting ourselves to a two-dimensional
plane. To separate the two classes of data points, there are many
possible lines (hyperplanes if you prefer a more strict naming)<br />
@@ -592,7 +527,7 @@ to divide the data. Below is an example of two candidate hyperplanes
for our data sample.</p>
</div>
<div class="section" id="getting-into-the-details">
<h2><span class="section-number">19.5. </span>Getting into the details<a class="headerlink" href="#getting-into-the-details" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.5. </span>Getting into the details<a class="headerlink" href="#getting-into-the-details" title="Permalink to this headline"></a></h2>
<p>Let us define the function</p>
<div class="math notranslate nohighlight">
\[
@@ -607,7 +542,7 @@ f(x) = \boldsymbol{w}^T\boldsymbol{x}+b = 0,
\]</div>
</div>
<div class="section" id="first-attempt-at-a-minimization-approach">
<h2><span class="section-number">19.6. </span>First attempt at a minimization approach<a class="headerlink" href="#first-attempt-at-a-minimization-approach" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.6. </span>First attempt at a minimization approach<a class="headerlink" href="#first-attempt-at-a-minimization-approach" title="Permalink to this headline"></a></h2>
<p>How do we find the parameter <span class="math notranslate nohighlight">\(b\)</span> and the vector <span class="math notranslate nohighlight">\(\boldsymbol{w}\)</span>? What we could
do is to define a cost function which now contains the set of all
misclassified points <span class="math notranslate nohighlight">\(M\)</span> and attempt to minimize this function</p>
@@ -627,7 +562,7 @@ C(\boldsymbol{w},b) = -\sum_{i\in M} y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b).
\]</div>
</div>
<div class="section" id="solving-the-equations">
<h2><span class="section-number">19.7. </span>Solving the equations<a class="headerlink" href="#solving-the-equations" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.7. </span>Solving the equations<a class="headerlink" href="#solving-the-equations" title="Permalink to this headline"></a></h2>
<p>We can now use the Newton-Raphson method or different variants of the gradient descent family (from plain gradient descent to various stochastic gradient descent approaches) to solve the equations</p>
<div class="math notranslate nohighlight">
\[
@@ -641,13 +576,13 @@ b \leftarrow b +\eta \frac{\partial C}{\partial b},
<p>where <span class="math notranslate nohighlight">\(\eta\)</span> is our by now well-known learning rate.</p>
</div>
<div class="section" id="code-example">
<h2><span class="section-number">19.8. </span>Code Example<a class="headerlink" href="#code-example" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.8. </span>Code Example<a class="headerlink" href="#code-example" title="Permalink to this headline"></a></h2>
<p>The equations we discussed above can be coded rather easily (the
framework is similar to what we developed for logistic
regression). We are going to set up a simple case with two classes only and we want to find a line which separates them the best possible way.</p>
</div>
<div class="section" id="problems-with-the-simpler-approach">
<h2><span class="section-number">19.9. </span>Problems with the Simpler Approach<a class="headerlink" href="#problems-with-the-simpler-approach" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.9. </span>Problems with the Simpler Approach<a class="headerlink" href="#problems-with-the-simpler-approach" title="Permalink to this headline"></a></h2>
<p>There are however problems with this approach, although it looks
pretty straightforward to implement. When running the above code, we see that we can easily end up with many diffeent lines which separate the two classes.</p>
<p>For small
@@ -657,7 +592,7 @@ properly into two distinct classes, we may not experience a converge
at all.</p>
</div>
<div class="section" id="a-better-approach">
<h2><span class="section-number">19.10. </span>A better approach<a class="headerlink" href="#a-better-approach" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.10. </span>A better approach<a class="headerlink" href="#a-better-approach" title="Permalink to this headline"></a></h2>
<p>A better approach is rather to try to define a large margin between
the two classes (if they are well separated from the beginning).</p>
<p>Thus, we wish to find a margin <span class="math notranslate nohighlight">\(M\)</span> with <span class="math notranslate nohighlight">\(\boldsymbol{w}\)</span> normalized to
@@ -689,7 +624,7 @@ possible margin <span class="math notranslate nohighlight">\(M\)</span>. Before
about Lagrangian multipliers.</p>
</div>
<div class="section" id="a-quick-reminder-on-lagrangian-multipliers">
<h2><span class="section-number">19.11. </span>A quick Reminder on Lagrangian Multipliers<a class="headerlink" href="#a-quick-reminder-on-lagrangian-multipliers" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.11. </span>A quick Reminder on Lagrangian Multipliers<a class="headerlink" href="#a-quick-reminder-on-lagrangian-multipliers" title="Permalink to this headline"></a></h2>
<p>Consider a function of three independent variables <span class="math notranslate nohighlight">\(f(x,y,z)\)</span> . For the function <span class="math notranslate nohighlight">\(f\)</span> to be an
extreme we have</p>
<div class="math notranslate nohighlight">
@@ -733,7 +668,7 @@ variables.
Then <span class="math notranslate nohighlight">\(dz\)</span> is no longer arbitrary.</p>
</div>
<div class="section" id="adding-the-multiplier">
<h2><span class="section-number">19.12. </span>Adding the Multiplier<a class="headerlink" href="#adding-the-multiplier" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.12. </span>Adding the Multiplier<a class="headerlink" href="#adding-the-multiplier" title="Permalink to this headline"></a></h2>
<p>However, we can add to</p>
<div class="math notranslate nohighlight">
\[
@@ -772,7 +707,7 @@ If we have a set of constraints <span class="math notranslate nohighlight">\(\ph
\]</div>
</div>
<div class="section" id="setting-up-the-problem">
<h2><span class="section-number">19.13. </span>Setting up the Problem<a class="headerlink" href="#setting-up-the-problem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.13. </span>Setting up the Problem<a class="headerlink" href="#setting-up-the-problem" title="Permalink to this headline"></a></h2>
<p>In order to solve the above problem, we define the following Lagrangian function to be minimized</p>
<div class="math notranslate nohighlight">
\[
@@ -807,7 +742,7 @@ We must in addition satisfy the <a class="reference external" href="https://en.w
<p>When <span class="math notranslate nohighlight">\(\lambda_i &gt; 0\)</span>, the vectors <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> are called support vectors. They are the vectors closest to the line (or hyperplane) and define the margin <span class="math notranslate nohighlight">\(M\)</span>.</p>
</div>
<div class="section" id="the-problem-to-solve">
<h2><span class="section-number">19.14. </span>The problem to solve<a class="headerlink" href="#the-problem-to-solve" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.14. </span>The problem to solve<a class="headerlink" href="#the-problem-to-solve" title="Permalink to this headline"></a></h2>
<p>We can rewrite</p>
<div class="math notranslate nohighlight">
\[
@@ -827,7 +762,7 @@ y_ny_1\boldsymbol{x}_n^T\boldsymbol{x}_1 &amp; y_ny_2\boldsymbol{x}_n^T\boldsymb
<span class="math notranslate nohighlight">\(\boldsymbol{y}=[y_1,y_2,\dots,y_n]\)</span>.</p>
</div>
<div class="section" id="the-last-steps">
<h2><span class="section-number">19.15. </span>The last steps<a class="headerlink" href="#the-last-steps" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.15. </span>The last steps<a class="headerlink" href="#the-last-steps" title="Permalink to this headline"></a></h2>
<p>Solving the above problem, yields the values of <span class="math notranslate nohighlight">\(\lambda_i\)</span>.
To find the coefficients of your hyperplane we need simply to compute</p>
<div class="math notranslate nohighlight">
@@ -857,7 +792,7 @@ y_i = \mathrm{sign}(\boldsymbol{w}^T\boldsymbol{x}_i+b).
<p>Below we discuss how to find the optimal values of <span class="math notranslate nohighlight">\(\lambda_i\)</span>. Before we proceed however, we discuss now the so-called soft classifier.</p>
</div>
<div class="section" id="a-soft-classifier">
<h2><span class="section-number">19.16. </span>A soft classifier<a class="headerlink" href="#a-soft-classifier" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.16. </span>A soft classifier<a class="headerlink" href="#a-soft-classifier" title="Permalink to this headline"></a></h2>
<p>Till now, the margin is strictly defined by the support vectors. This defines what is called a hard classifier, that is the margins are well defined.</p>
<p>Suppose now that classes overlap in feature space, as shown in the
figure here. One way to deal with this problem before we define the
@@ -882,7 +817,7 @@ we bound the total amount by which predictions fall on the wrong side of their m
misclassifications.</p>
</div>
<div class="section" id="soft-optmization-problem">
<h2><span class="section-number">19.17. </span>Soft optmization problem<a class="headerlink" href="#soft-optmization-problem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.17. </span>Soft optmization problem<a class="headerlink" href="#soft-optmization-problem" title="Permalink to this headline"></a></h2>
<p>This has in turn the consequences that we change our optmization problem to finding the minimum of</p>
<div class="math notranslate nohighlight">
\[
@@ -944,7 +879,7 @@ y_i(\boldsymbol{w}^T\boldsymbol{x}_i+b) -(1-\xi_) \geq 0 \hspace{0.1cm}\forall i
\]</div>
</div>
<div class="section" id="kernels-and-non-linearity">
<h2><span class="section-number">19.18. </span>Kernels and non-linearity<a class="headerlink" href="#kernels-and-non-linearity" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.18. </span>Kernels and non-linearity<a class="headerlink" href="#kernels-and-non-linearity" title="Permalink to this headline"></a></h2>
<p>The cases we have studied till now, were all characterized by two classes
with a close to linear separability. The classifiers we have described
so far find linear boundaries in our input feature space. It is
@@ -1014,7 +949,7 @@ we need to introduce for example a polynomial transformation to a two-dimensiona
</div>
</div>
<div class="section" id="the-equations">
<h2><span class="section-number">19.19. </span>The equations<a class="headerlink" href="#the-equations" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.19. </span>The equations<a class="headerlink" href="#the-equations" title="Permalink to this headline"></a></h2>
<p>Suppose we define a polynomial transformation of degree two only (we continue to live in a plane with <span class="math notranslate nohighlight">\(x_i\)</span> and <span class="math notranslate nohighlight">\(y_i\)</span> as variables)</p>
<div class="math notranslate nohighlight">
\[
@@ -1051,7 +986,7 @@ the trouble of performing the transformation
<span class="math notranslate nohighlight">\(\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}_j)\)</span> during the SVM calculations.</p>
</div>
<div class="section" id="id1">
<h2><span class="section-number">19.20. </span>The problem to solve<a class="headerlink" href="#id1" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.20. </span>The problem to solve<a class="headerlink" href="#id1" title="Permalink to this headline"></a></h2>
<p>Using our definition of the kernel We can rewrite again the Lagrangian</p>
<div class="math notranslate nohighlight">
\[
@@ -1083,7 +1018,7 @@ Given a kernel <span class="math notranslate nohighlight">\(K\)</span> and the t
<span class="math notranslate nohighlight">\(0\leq \lambda_i\)</span> and <span class="math notranslate nohighlight">\(\lambda_i \leq C\)</span>. These two inequalities define then the matrix <span class="math notranslate nohighlight">\(\boldsymbol{G}\)</span> and the vector <span class="math notranslate nohighlight">\(\boldsymbol{h}\)</span>.</p>
</div>
<div class="section" id="different-kernels-and-mercer-s-theorem">
<h2><span class="section-number">19.21. </span>Different kernels and Mercer’s theorem<a class="headerlink" href="#different-kernels-and-mercer-s-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.21. </span>Different kernels and Mercer’s theorem<a class="headerlink" href="#different-kernels-and-mercer-s-theorem" title="Permalink to this headline"></a></h2>
<p>There are several popular kernels being used. These are</p>
<ol class="simple">
<li><p>Linear: <span class="math notranslate nohighlight">\(K(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{x}^T\boldsymbol{y}\)</span>,</p></li>
@@ -1109,7 +1044,7 @@ don’t respect all of Mercer’s conditions, yet they generally work well
in practice.</p>
</div>
<div class="section" id="the-moons-example">
<h2><span class="section-number">19.22. </span>The moons example<a class="headerlink" href="#the-moons-example" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.22. </span>The moons example<a class="headerlink" href="#the-moons-example" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">division</span><span class="p">,</span> <span class="n">print_function</span><span class="p">,</span> <span class="n">unicode_literals</span>
@@ -1321,7 +1256,7 @@ Phi(-1.0, 1) = [0.30119421]
</div>
</div>
<div class="section" id="mathematical-optimization-of-convex-functions">
<h2><span class="section-number">19.23. </span>Mathematical optimization of convex functions<a class="headerlink" href="#mathematical-optimization-of-convex-functions" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.23. </span>Mathematical optimization of convex functions<a class="headerlink" href="#mathematical-optimization-of-convex-functions" title="Permalink to this headline"></a></h2>
<p>A mathematical (quadratic) optimization problem, or just optimization problem, has the form</p>
<div class="math notranslate nohighlight">
\[\begin{split}
@@ -1338,7 +1273,7 @@ In our discussion on gradient descent methods we discussed at length the definit
<p>Convex optimization problems play a central role in applied mathematics and we recommend strongly <a class="reference external" href="http://web.stanford.edu/~boyd/cvxbook/">Boyd and Vandenberghe’s text on the topics</a>.</p>
</div>
<div class="section" id="how-do-we-solve-these-problems">
<h2><span class="section-number">19.24. </span>How do we solve these problems?<a class="headerlink" href="#how-do-we-solve-these-problems" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.24. </span>How do we solve these problems?<a class="headerlink" href="#how-do-we-solve-these-problems" title="Permalink to this headline"></a></h2>
<p>If we use Python as programming language and wish to venture beyond
<strong>scikit-learn</strong>, <strong>tensorflow</strong> and similar software which makes our
lives so much easier, we need to dive into the wonderful world of
@@ -1358,7 +1293,7 @@ converge. So, welcome to the promised land of quadratic programming.</p>
<p>This will make our life much easier. You don’t need t write your own optimizer.</p>
</div>
<div class="section" id="a-simple-example">
<h2><span class="section-number">19.25. </span>A simple example<a class="headerlink" href="#a-simple-example" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.25. </span>A simple example<a class="headerlink" href="#a-simple-example" title="Permalink to this headline"></a></h2>
<p>We remind ourselves about the general problem we want to solve</p>
<div class="math notranslate nohighlight">
\[\begin{split}
@@ -1431,7 +1366,7 @@ SyntaxError: invalid character in identifier
</div>
</div>
<div class="section" id="back-to-the-more-realistic-cases">
<h2><span class="section-number">19.26. </span>Back to the more realistic cases<a class="headerlink" href="#back-to-the-more-realistic-cases" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">6.26. </span>Back to the more realistic cases<a class="headerlink" href="#back-to-the-more-realistic-cases" title="Permalink to this headline"></a></h2>
<p>We are now ready to return to our setup of the optmization problem for a more realistic case. Introducing the <strong>slack</strong> parameter <span class="math notranslate nohighlight">\(C\)</span> we have</p>
<div class="math notranslate nohighlight">
\[\begin{split}
@@ -1477,8 +1412,8 @@ With the slack constants this leads to the additional constraint <span class="m
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter6.html" title="previous page"><span class="section-number">18. </span>Neural networks, from the simple perceptron to deep learning</a>
<a class='right-next' id="next-link" href="chapter8.html" title="next page"><span class="section-number">20. </span>Dimensionality Reduction</a>
<a class='left-prev' id="prev-link" href="chapter6.html" title="previous page"><span class="section-number">5. </span>Neural networks, from the simple perceptron to deep learning</a>
<a class='right-next' id="next-link" href="chapter8.html" title="next page"><span class="section-number">7. </span>Dimensionality Reduction</a>
</div>
<footer class="footer mt-5 mt-md-0">
doc/src/LectureNotes/_build/html/chapter8.html
+80 -145
View File
@@ -6,7 +6,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>20. Dimensionality Reduction &#8212; Applied Machine Learning and Data Analysis</title>
<title>7. Dimensionality Reduction &#8212; Applied Machine Learning and Data Analysis</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.11.2/css/all.min.css" integrity="sha384-KA6wR/X5RY4zFAHpv/CnoG2UW1uogYfdnP67Uv7eULvTveboZJg0qUpmJZb5VqzN" crossorigin="anonymous">
<link href="_static/css/index.css" rel="stylesheet">
<link rel="stylesheet" href="_static/sphinx-book-theme.css" type="text/css" />
@@ -38,7 +38,7 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="prev" title="19. Support Vector Machines, overarching aims" href="chapter7.html" />
<link rel="prev" title="6. Support Vector Machines, overarching aims" href="chapter7.html" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="docsearch:language" content="en">
@@ -89,99 +89,34 @@
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers">
2. Random Numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers-better-name-pseudo-random-numbers">
3. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng">
4. Random number generator RNG
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-periodic-outputs">
5. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-its-period">
6. Random number generator RNG and its period
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-other-examples">
7. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id9">
8. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0">
9. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id10">
10. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id11">
11. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id12">
12. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0-code">
13. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#which-rng-should-i-use">
14. Which RNG should I use?
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
15. Getting started, our first data and Machine Learning encounters
2. Getting started, our first data and Machine Learning encounters
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
16. Linear Regression and more Advanced Regression Analysis
3. Linear Regression and more Advanced Regression Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
17. Logistic Regression
4. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
18. Neural networks, from the simple perceptron to deep learning
5. Neural networks, from the simple perceptron to deep learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
19. Support Vector Machines, overarching aims
6. Support Vector Machines, overarching aims
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
20. Dimensionality Reduction
7. Dimensionality Reduction
</a>
</li>
</ul>
@@ -256,179 +191,179 @@
<ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">
20.1. Reducing the number of degrees of freedom, overarching view
7.1. Reducing the number of degrees of freedom, overarching view
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#preprocessing-our-data">
20.2. Preprocessing our data
7.2. Preprocessing our data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#more-preprocessing">
20.3. More preprocessing
7.3. More preprocessing
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-preprocessing-examples-franke-function-and-regression">
20.4. Simple preprocessing examples, Franke function and regression
7.4. Simple preprocessing examples, Franke function and regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-preprocessing-examples-breast-cancer-data-and-classification-support-vector-machines">
20.5. Simple preprocessing examples, breast cancer data and classification, Support Vector Machines
7.5. Simple preprocessing examples, breast cancer data and classification, Support Vector Machines
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#more-on-cancer-data-now-with-logistic-regression">
20.6. More on Cancer Data, now with Logistic Regression
7.6. More on Cancer Data, now with Logistic Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#why-should-we-think-of-reducing-the-dimensionality">
20.7. Why should we think of reducing the dimensionality
7.7. Why should we think of reducing the dimensionality
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-ideas-of-the-principal-component-analysis-pca">
20.8. Basic ideas of the Principal Component Analysis (PCA)
7.8. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#introducing-the-covariance-and-correlation-functions">
20.9. Introducing the Covariance and Correlation functions
7.9. Introducing the Covariance and Correlation functions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-function-and-design-feature-matrix">
20.10. Correlation Function and Design/Feature Matrix
7.10. Correlation Function and Design/Feature Matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#covariance-matrix-examples">
20.11. Covariance Matrix Examples
7.11. Covariance Matrix Examples
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-matrix">
20.12. Correlation Matrix
7.12. Correlation Matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-matrix-with-pandas">
20.13. Correlation Matrix with Pandas
7.13. Correlation Matrix with Pandas
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-matrix-with-pandas-and-the-franke-function">
20.14. Correlation Matrix with Pandas and the Franke function
7.14. Correlation Matrix with Pandas and the Franke function
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#rewriting-the-covariance-and-or-correlation-matrix">
20.15. Rewriting the Covariance and/or Correlation Matrix
7.15. Rewriting the Covariance and/or Correlation Matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#towards-the-pca-theorem">
20.16. Towards the PCA theorem
7.16. Towards the PCA theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-algorithm-before-theorem">
20.17. The Algorithm before theorem
7.17. The Algorithm before theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#writing-our-own-pca-code">
20.18. Writing our own PCA code
7.18. Writing our own PCA code
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#compute-the-sample-mean-and-center-the-data">
20.18.1. Compute the sample mean and center the data
7.18.1. Compute the sample mean and center the data
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#compute-the-sample-covariance">
20.18.2. Compute the sample covariance
7.18.2. Compute the sample covariance
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">
20.18.3. Diagonalize the sample covariance matrix to obtain the principal components
7.18.3. Diagonalize the sample covariance matrix to obtain the principal components
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#classical-pca-theorem">
20.19. Classical PCA Theorem
7.19. Classical PCA Theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#proof-of-the-pca-theorem">
20.20. Proof of the PCA Theorem
7.20. Proof of the PCA Theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pca-proof-continued">
20.21. PCA Proof continued
7.21. PCA Proof continued
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-final-step">
20.22. The final step
7.22. The final step
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#geometric-interpretation-and-link-with-singular-value-decomposition">
20.23. Geometric Interpretation and link with Singular Value Decomposition
7.23. Geometric Interpretation and link with Singular Value Decomposition
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#principal-component-analysis">
20.24. Principal Component Analysis
7.24. Principal Component Analysis
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pca-and-scikit-learn">
20.25. PCA and scikit-learn
7.25. PCA and scikit-learn
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#back-to-the-cancer-data">
20.26. Back to the Cancer Data
7.26. Back to the Cancer Data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#more-on-the-pca">
20.27. More on the PCA
7.27. More on the PCA
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#incremental-pca">
20.28. Incremental PCA
7.28. Incremental PCA
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#randomized-pca">
20.29. Randomized PCA
7.29. Randomized PCA
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#kernel-pca">
20.30. Kernel PCA
7.30. Kernel PCA
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#lle">
20.31. LLE
7.31. LLE
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-techniques">
20.32. Other techniques
7.32. Other techniques
</a>
</li>
</ul>
@@ -442,9 +377,9 @@
<div>
<div class="section" id="dimensionality-reduction">
<h1><span class="section-number">20. </span>Dimensionality Reduction<a class="headerlink" href="#dimensionality-reduction" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">7. </span>Dimensionality Reduction<a class="headerlink" href="#dimensionality-reduction" title="Permalink to this headline"></a></h1>
<div class="section" id="reducing-the-number-of-degrees-of-freedom-overarching-view">
<h2><span class="section-number">20.1. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.1. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline"></a></h2>
<p>Many Machine Learning problems involve thousands or even millions of
features for each training instance. Not only does this make training
extremely slow, it can also make it much harder to find a good
@@ -465,7 +400,7 @@ is one of the most used tools in data modeling, compression and
visualization.</p>
</div>
<div class="section" id="preprocessing-our-data">
<h2><span class="section-number">20.2. </span>Preprocessing our data<a class="headerlink" href="#preprocessing-our-data" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.2. </span>Preprocessing our data<a class="headerlink" href="#preprocessing-our-data" title="Permalink to this headline"></a></h2>
<p>Before we proceed however, we will discuss how to preprocess our
data. Till now and in connection with our previous examples we have
not met so many cases where we are too sensitive to the scaling of our
@@ -483,7 +418,7 @@ function included in <strong>Scikit-Learn</strong> is the <strong>MinMaxScaler</
ensures that all features are exactly between <span class="math notranslate nohighlight">\(0\)</span> and <span class="math notranslate nohighlight">\(1\)</span>. The</p>
</div>
<div class="section" id="more-preprocessing">
<h2><span class="section-number">20.3. </span>More preprocessing<a class="headerlink" href="#more-preprocessing" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.3. </span>More preprocessing<a class="headerlink" href="#more-preprocessing" title="Permalink to this headline"></a></h2>
<p>The <strong>Normalizer</strong> scales each data
point such that the feature vector has a euclidean length of one. In other words, it
projects a data point on the circle (or sphere in the case of higher dimensions) with a
@@ -501,7 +436,7 @@ outliers, and might often lead to trouble for other scaling
techniques.</p>
</div>
<div class="section" id="simple-preprocessing-examples-franke-function-and-regression">
<h2><span class="section-number">20.4. </span>Simple preprocessing examples, Franke function and regression<a class="headerlink" href="#simple-preprocessing-examples-franke-function-and-regression" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.4. </span>Simple preprocessing examples, Franke function and regression<a class="headerlink" href="#simple-preprocessing-examples-franke-function-and-regression" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
@@ -636,7 +571,7 @@ R2 score for scaled data: 0.97
</div>
</div>
<div class="section" id="simple-preprocessing-examples-breast-cancer-data-and-classification-support-vector-machines">
<h2><span class="section-number">20.5. </span>Simple preprocessing examples, breast cancer data and classification, Support Vector Machines<a class="headerlink" href="#simple-preprocessing-examples-breast-cancer-data-and-classification-support-vector-machines" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.5. </span>Simple preprocessing examples, breast cancer data and classification, Support Vector Machines<a class="headerlink" href="#simple-preprocessing-examples-breast-cancer-data-and-classification-support-vector-machines" title="Permalink to this headline"></a></h2>
<p>We show here how we can use a simple regression case on the breast
cancer data using support vector machines (SVM) as algorithm for
classification.</p>
@@ -715,7 +650,7 @@ Test set accuracy scaled data with Standar Scaler: 0.96
</div>
</div>
<div class="section" id="more-on-cancer-data-now-with-logistic-regression">
<h2><span class="section-number">20.6. </span>More on Cancer Data, now with Logistic Regression<a class="headerlink" href="#more-on-cancer-data-now-with-logistic-regression" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.6. </span>More on Cancer Data, now with Logistic Regression<a class="headerlink" href="#more-on-cancer-data-now-with-logistic-regression" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
@@ -761,7 +696,7 @@ Please also refer to the documentation for alternative solver options:
</div>
</div>
<div class="section" id="why-should-we-think-of-reducing-the-dimensionality">
<h2><span class="section-number">20.7. </span>Why should we think of reducing the dimensionality<a class="headerlink" href="#why-should-we-think-of-reducing-the-dimensionality" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.7. </span>Why should we think of reducing the dimensionality<a class="headerlink" href="#why-should-we-think-of-reducing-the-dimensionality" title="Permalink to this headline"></a></h2>
<p>In addition to the plot of the features, we study now also the covariance (and the correlation matrix).
We use also <strong>Pandas</strong> to compute the correlation matrix.</p>
<div class="cell docutils container">
@@ -854,7 +789,7 @@ the classical Principal Component Analysis (PCA) theorem with
applications.</p>
</div>
<div class="section" id="basic-ideas-of-the-principal-component-analysis-pca">
<h2><span class="section-number">20.8. </span>Basic ideas of the Principal Component Analysis (PCA)<a class="headerlink" href="#basic-ideas-of-the-principal-component-analysis-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.8. </span>Basic ideas of the Principal Component Analysis (PCA)<a class="headerlink" href="#basic-ideas-of-the-principal-component-analysis-pca" title="Permalink to this headline"></a></h2>
<p>The principal component analysis deals with the problem of fitting a
low-dimensional affine subspace <span class="math notranslate nohighlight">\(S\)</span> of dimension <span class="math notranslate nohighlight">\(d\)</span> much smaller than
the totaldimension <span class="math notranslate nohighlight">\(D\)</span> of the problem at hand (our data
@@ -870,7 +805,7 @@ what set the scene historically which for the PCA.</p>
</ul>
</div>
<div class="section" id="introducing-the-covariance-and-correlation-functions">
<h2><span class="section-number">20.9. </span>Introducing the Covariance and Correlation functions<a class="headerlink" href="#introducing-the-covariance-and-correlation-functions" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.9. </span>Introducing the Covariance and Correlation functions<a class="headerlink" href="#introducing-the-covariance-and-correlation-functions" title="Permalink to this headline"></a></h2>
<p>Before we discuss the PCA theorem, we need to remind ourselves about
the definition of the covariance and the correlation function. These are quantities</p>
<p>Suppose we have defined two vectors
@@ -920,7 +855,7 @@ and <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as</p>
<p>In the above example this is the function we constructed using <strong>pandas</strong>.</p>
</div>
<div class="section" id="correlation-function-and-design-feature-matrix">
<h2><span class="section-number">20.10. </span>Correlation Function and Design/Feature Matrix<a class="headerlink" href="#correlation-function-and-design-feature-matrix" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.10. </span>Correlation Function and Design/Feature Matrix<a class="headerlink" href="#correlation-function-and-design-feature-matrix" title="Permalink to this headline"></a></h2>
<p>In our derivation of the various regression algorithms like <strong>Ordinary Least Squares</strong> or <strong>Ridge regression</strong>
we defined the design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> as</p>
<div class="math notranslate nohighlight">
@@ -975,7 +910,7 @@ covariance matrix for the vectors <span class="math notranslate nohighlight">\(\
\end{split}\]</div>
</div>
<div class="section" id="covariance-matrix-examples">
<h2><span class="section-number">20.11. </span>Covariance Matrix Examples<a class="headerlink" href="#covariance-matrix-examples" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.11. </span>Covariance Matrix Examples<a class="headerlink" href="#covariance-matrix-examples" title="Permalink to this headline"></a></h2>
<p>The Numpy function <strong>np.cov</strong> calculates the covariance elements using
the factor <span class="math notranslate nohighlight">\(1/(n-1)\)</span> instead of <span class="math notranslate nohighlight">\(1/n\)</span> since it assumes we do not have
the exact mean values. The following simple function uses the
@@ -1022,7 +957,7 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
</div>
</div>
<div class="section" id="correlation-matrix">
<h2><span class="section-number">20.12. </span>Correlation Matrix<a class="headerlink" href="#correlation-matrix" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.12. </span>Correlation Matrix<a class="headerlink" href="#correlation-matrix" title="Permalink to this headline"></a></h2>
<p>The previous example can be converted into the correlation matrix by
simply scaling the matrix elements with the variances. We should also
subtract the mean values for each column. This leads to the following
@@ -1069,7 +1004,7 @@ this matrix we easily see that it is a positive definite matrix.</p>
<p>The above procedure with <strong>numpy</strong> can be made more compact if we use <strong>pandas</strong>.</p>
</div>
<div class="section" id="correlation-matrix-with-pandas">
<h2><span class="section-number">20.13. </span>Correlation Matrix with Pandas<a class="headerlink" href="#correlation-matrix-with-pandas" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.13. </span>Correlation Matrix with Pandas<a class="headerlink" href="#correlation-matrix-with-pandas" title="Permalink to this headline"></a></h2>
<p>We whow here how we can set up the correlation matrix using <strong>pandas</strong>, as done in this simple code</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -1121,7 +1056,7 @@ this matrix we easily see that it is a positive definite matrix.</p>
<p>We expand this model to the Franke function discussed above.</p>
</div>
<div class="section" id="correlation-matrix-with-pandas-and-the-franke-function">
<h2><span class="section-number">20.14. </span>Correlation Matrix with Pandas and the Franke function<a class="headerlink" href="#correlation-matrix-with-pandas-and-the-franke-function" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.14. </span>Correlation Matrix with Pandas and the Franke function<a class="headerlink" href="#correlation-matrix-with-pandas-and-the-franke-function" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
@@ -1217,7 +1152,7 @@ drop these elements and construct a correlation
matrix without these elements.</p>
</div>
<div class="section" id="rewriting-the-covariance-and-or-correlation-matrix">
<h2><span class="section-number">20.15. </span>Rewriting the Covariance and/or Correlation Matrix<a class="headerlink" href="#rewriting-the-covariance-and-or-correlation-matrix" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.15. </span>Rewriting the Covariance and/or Correlation Matrix<a class="headerlink" href="#rewriting-the-covariance-and-or-correlation-matrix" title="Permalink to this headline"></a></h2>
<p>We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
@@ -1252,7 +1187,7 @@ x_{10}x_{00}+x_{11}x_{01} &amp; x_{10}^2+x_{11}^2\\
<p>It is easy to generalize this to a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>.</p>
</div>
<div class="section" id="towards-the-pca-theorem">
<h2><span class="section-number">20.16. </span>Towards the PCA theorem<a class="headerlink" href="#towards-the-pca-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.16. </span>Towards the PCA theorem<a class="headerlink" href="#towards-the-pca-theorem" title="Permalink to this headline"></a></h2>
<p>We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as</p>
<div class="math notranslate nohighlight">
\[
@@ -1291,7 +1226,7 @@ we could then aim at reducing <span class="math notranslate nohighlight">\(p\)</
features/predictors.</p>
</div>
<div class="section" id="the-algorithm-before-theorem">
<h2><span class="section-number">20.17. </span>The Algorithm before theorem<a class="headerlink" href="#the-algorithm-before-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.17. </span>The Algorithm before theorem<a class="headerlink" href="#the-algorithm-before-theorem" title="Permalink to this headline"></a></h2>
<p>Here’s how we would proceed in setting up the algorithm for the PCA, see also discussion below here.</p>
<ul class="simple">
<li><p>Set up the datapoints for the design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> with <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>, with the predictors/features <span class="math notranslate nohighlight">\(p\)</span> referring to the column numbers and the entries <span class="math notranslate nohighlight">\(n\)</span> being the row elements.</p></li>
@@ -1316,7 +1251,7 @@ x_{n-1,0} &amp; x_{n-1,1} &amp; x_{n-1,2}&amp; \dots &amp; \dots x_{n-1,p-1}\\
</ul>
</div>
<div class="section" id="writing-our-own-pca-code">
<h2><span class="section-number">20.18. </span>Writing our own PCA code<a class="headerlink" href="#writing-our-own-pca-code" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.18. </span>Writing our own PCA code<a class="headerlink" href="#writing-our-own-pca-code" title="Permalink to this headline"></a></h2>
<p>We will use a simple example first with two-dimensional data
drawn from a multivariate normal distribution with the following mean and covariance matrix:</p>
<div class="math notranslate nohighlight">
@@ -1346,7 +1281,7 @@ Note that the function <strong>multivariate</strong> returns also the covariance
</div>
<p>Now we are going to implement the PCA algorithm. We will break it down into various substeps.</p>
<div class="section" id="compute-the-sample-mean-and-center-the-data">
<h3><span class="section-number">20.18.1. </span>Compute the sample mean and center the data<a class="headerlink" href="#compute-the-sample-mean-and-center-the-data" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">7.18.1. </span>Compute the sample mean and center the data<a class="headerlink" href="#compute-the-sample-mean-and-center-the-data" title="Permalink to this headline"></a></h3>
<p>The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is</p>
<div class="math notranslate nohighlight">
\[
@@ -1383,7 +1318,7 @@ while the non-diagonal ones need to be divided by <span class="math notranslate
specific case.</p>
</div>
<div class="section" id="compute-the-sample-covariance">
<h3><span class="section-number">20.18.2. </span>Compute the sample covariance<a class="headerlink" href="#compute-the-sample-covariance" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">7.18.2. </span>Compute the sample covariance<a class="headerlink" href="#compute-the-sample-covariance" title="Permalink to this headline"></a></h3>
<p>Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation</p>
<div class="math notranslate nohighlight">
\[
@@ -1441,7 +1376,7 @@ Our own code here is not very elegant and asks for obvious improvements. It is t
The plot shows how the data are clustered around a line with slope close to one. Is this expected?</p>
</div>
<div class="section" id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">
<h3><span class="section-number">20.18.3. </span>Diagonalize the sample covariance matrix to obtain the principal components<a class="headerlink" href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">7.18.3. </span>Diagonalize the sample covariance matrix to obtain the principal components<a class="headerlink" href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" title="Permalink to this headline"></a></h3>
<p>Now we are ready to solve for the principal components! To do so we
diagonalize the sample covariance matrix <span class="math notranslate nohighlight">\(\Sigma\)</span>. We can use the
function <strong>np.linalg.eig</strong> to do so. It will return the eigenvalues and
@@ -1507,7 +1442,7 @@ Eigenvector of largest eigenvalue
</div>
</div>
<div class="section" id="classical-pca-theorem">
<h2><span class="section-number">20.19. </span>Classical PCA Theorem<a class="headerlink" href="#classical-pca-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.19. </span>Classical PCA Theorem<a class="headerlink" href="#classical-pca-theorem" title="Permalink to this headline"></a></h2>
<p>We assume now that we have a design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which has been
centered as discussed above. For the sake of simplicity we skip the
overline symbol. The matrix is defined in terms of the various column
@@ -1530,7 +1465,7 @@ eigenvectors of the covariance(correlations matrix).</p>
<p>The proof which follows will be updated by mid January 2020.</p>
</div>
<div class="section" id="proof-of-the-pca-theorem">
<h2><span class="section-number">20.20. </span>Proof of the PCA Theorem<a class="headerlink" href="#proof-of-the-pca-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.20. </span>Proof of the PCA Theorem<a class="headerlink" href="#proof-of-the-pca-theorem" title="Permalink to this headline"></a></h2>
<p>To show the PCA theorem let us start with the assumption that there is one vector <span class="math notranslate nohighlight">\(\boldsymbol{w}_0\)</span> which corresponds to a solution which minimized the reconstruction error <span class="math notranslate nohighlight">\(J\)</span>. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of <span class="math notranslate nohighlight">\(\boldsymbol{w}_0\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{z}_0\)</span> as</p>
<div class="math notranslate nohighlight">
\[
@@ -1549,7 +1484,7 @@ z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i,
<p>where the vectors on the rhs are known.</p>
</div>
<div class="section" id="pca-proof-continued">
<h2><span class="section-number">20.21. </span>PCA Proof continued<a class="headerlink" href="#pca-proof-continued" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.21. </span>PCA Proof continued<a class="headerlink" href="#pca-proof-continued" title="Permalink to this headline"></a></h2>
<p>We have now found the unknown parameters <span class="math notranslate nohighlight">\(z_{i0}\)</span>. These correspond to the projected coordinates and we can write</p>
<div class="math notranslate nohighlight">
\[
@@ -1582,7 +1517,7 @@ matrix. Minimizing the error is equivalent to maximizing the variance
of the projected data.</p>
</div>
<div class="section" id="the-final-step">
<h2><span class="section-number">20.22. </span>The final step<a class="headerlink" href="#the-final-step" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.22. </span>The final step<a class="headerlink" href="#the-final-step" title="Permalink to this headline"></a></h2>
<p>We could trivially maximize the variance of the projection (and
thereby minimize the error in the reconstruction function) by letting
the norm-2 of <span class="math notranslate nohighlight">\(\boldsymbol{w}_0\)</span> go to infinity. However, this norm since we
@@ -1625,11 +1560,11 @@ chapter 12.4 and discussion therein.</p>
<p>Additional part of the proof for the other eigenvectors will be added by mid January 2020.</p>
</div>
<div class="section" id="geometric-interpretation-and-link-with-singular-value-decomposition">
<h2><span class="section-number">20.23. </span>Geometric Interpretation and link with Singular Value Decomposition<a class="headerlink" href="#geometric-interpretation-and-link-with-singular-value-decomposition" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.23. </span>Geometric Interpretation and link with Singular Value Decomposition<a class="headerlink" href="#geometric-interpretation-and-link-with-singular-value-decomposition" title="Permalink to this headline"></a></h2>
<p>This material will be added by mid January 2020.</p>
</div>
<div class="section" id="principal-component-analysis">
<h2><span class="section-number">20.24. </span>Principal Component Analysis<a class="headerlink" href="#principal-component-analysis" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.24. </span>Principal Component Analysis<a class="headerlink" href="#principal-component-analysis" title="Permalink to this headline"></a></h2>
<p>Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.</p>
<p>The following Python code uses NumPy’s <strong>svd()</strong> function to obtain all the principal components of the
@@ -1813,7 +1748,7 @@ Selecting this hyperplane ensures that the projection will preserve as much vari
</div>
</div>
<div class="section" id="pca-and-scikit-learn">
<h2><span class="section-number">20.25. </span>PCA and scikit-learn<a class="headerlink" href="#pca-and-scikit-learn" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.25. </span>PCA and scikit-learn<a class="headerlink" href="#pca-and-scikit-learn" title="Permalink to this headline"></a></h2>
<p>Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
that it automatically takes care of centering the data):</p>
@@ -1865,7 +1800,7 @@ available via the <span class="math notranslate nohighlight">\(explained\_varian
variance that lies along the axis of each principal component.</p>
</div>
<div class="section" id="back-to-the-cancer-data">
<h2><span class="section-number">20.26. </span>Back to the Cancer Data<a class="headerlink" href="#back-to-the-cancer-data" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.26. </span>Back to the Cancer Data<a class="headerlink" href="#back-to-the-cancer-data" title="Permalink to this headline"></a></h2>
<p>We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.</p>
<div class="cell docutils container">
@@ -1905,7 +1840,7 @@ Here we compute performance scores on the training data using logistic regressio
<p>We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.</p>
</div>
<div class="section" id="more-on-the-pca">
<h2><span class="section-number">20.27. </span>More on the PCA<a class="headerlink" href="#more-on-the-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.27. </span>More on the PCA<a class="headerlink" href="#more-on-the-pca" title="Permalink to this headline"></a></h2>
<p>Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
Unless, of course, you are reducing dimensionality for data visualization — in that case you will
@@ -1935,7 +1870,7 @@ a float between 0.0 and 1.0, indicating the ratio of variance you wish to preser
</div>
</div>
<div class="section" id="incremental-pca">
<h2><span class="section-number">20.28. </span>Incremental PCA<a class="headerlink" href="#incremental-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.28. </span>Incremental PCA<a class="headerlink" href="#incremental-pca" title="Permalink to this headline"></a></h2>
<p>One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
@@ -1943,14 +1878,14 @@ at a time. This is useful for large training sets, and also to apply PCA online
instances arrive).</p>
</div>
<div class="section" id="randomized-pca">
<h2><span class="section-number">20.29. </span>Randomized PCA<a class="headerlink" href="#randomized-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.29. </span>Randomized PCA<a class="headerlink" href="#randomized-pca" title="Permalink to this headline"></a></h2>
<p>Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
algorithm that quickly finds an approximation of the first d principal components. Its computational
complexity is <span class="math notranslate nohighlight">\(O(m \times d^2)+O(d^3)\)</span>, instead of <span class="math notranslate nohighlight">\(O(m \times n^2) + O(n^3)\)</span>, so it is dramatically faster than the
previous algorithms when <span class="math notranslate nohighlight">\(d\)</span> is much smaller than <span class="math notranslate nohighlight">\(n\)</span>.</p>
</div>
<div class="section" id="kernel-pca">
<h2><span class="section-number">20.30. </span>Kernel PCA<a class="headerlink" href="#kernel-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.30. </span>Kernel PCA<a class="headerlink" href="#kernel-pca" title="Permalink to this headline"></a></h2>
<p>The kernel trick is a mathematical technique that implicitly maps instances into a
very high-dimensional space (called the feature space), enabling nonlinear classification and regression
with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
@@ -1971,7 +1906,7 @@ For example, the following code uses Scikit-Learn’s KernelPCA class to perform
</div>
</div>
<div class="section" id="lle">
<h2><span class="section-number">20.31. </span>LLE<a class="headerlink" href="#lle" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.31. </span>LLE<a class="headerlink" href="#lle" title="Permalink to this headline"></a></h2>
<p>Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
@@ -1979,7 +1914,7 @@ closest neighbors (c.n.), and then looking for a low-dimensional representation
these local relationships are best preserved (more details shortly).</p>
</div>
<div class="section" id="other-techniques">
<h2><span class="section-number">20.32. </span>Other techniques<a class="headerlink" href="#other-techniques" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.32. </span>Other techniques<a class="headerlink" href="#other-techniques" title="Permalink to this headline"></a></h2>
<p>There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.</p>
<p>Here are some of the most popular:</p>
<ul class="simple">
@@ -2019,7 +1954,7 @@ these local relationships are best preserved (more details shortly).</p>
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter7.html" title="previous page"><span class="section-number">19. </span>Support Vector Machines, overarching aims</a>
<a class='left-prev' id="prev-link" href="chapter7.html" title="previous page"><span class="section-number">6. </span>Support Vector Machines, overarching aims</a>
</div>
<footer class="footer mt-5 mt-md-0">
doc/src/LectureNotes/_build/html/genindex.html
+6 -71
View File
@@ -88,99 +88,34 @@
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers">
2. Random Numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers-better-name-pseudo-random-numbers">
3. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng">
4. Random number generator RNG
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-periodic-outputs">
5. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-its-period">
6. Random number generator RNG and its period
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-other-examples">
7. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id9">
8. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0">
9. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id10">
10. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id11">
11. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id12">
12. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0-code">
13. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#which-rng-should-i-use">
14. Which RNG should I use?
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
15. Getting started, our first data and Machine Learning encounters
2. Getting started, our first data and Machine Learning encounters
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
16. Linear Regression and more Advanced Regression Analysis
3. Linear Regression and more Advanced Regression Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
17. Logistic Regression
4. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
18. Neural networks, from the simple perceptron to deep learning
5. Neural networks, from the simple perceptron to deep learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
19. Support Vector Machines, overarching aims
6. Support Vector Machines, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
20. Dimensionality Reduction
7. Dimensionality Reduction
</a>
</li>
</ul>
doc/src/LectureNotes/_build/html/reports/chapter2.log
+34 -7
View File
@@ -17,15 +17,42 @@ Traceback (most recent call last):
raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
import numpy as np
x = np.log(np.array([4.0, 7.0, 8.0])
print(x)
# Blocking
@timeFunction
def blocking(self, blockSizeMax = 500):
blockSizeMin = 1
self.blockSizes = []
self.meanVec = []
self.varVec = []
for i in range(blockSizeMin, blockSizeMax):
if(len(self.data) % i != 0):
pass#continue
blockSize = i
meanTempVec = []
varTempVec = []
startPoint = 0
endPoint = blockSize
while endPoint <= len(self.data):
meanTempVec.append(np.average(self.data[startPoint:endPoint]))
startPoint = endPoint
endPoint += blockSize
mean, var = np.average(meanTempVec), np.var(meanTempVec)/len(meanTempVec)
self.meanVec.append(mean)
self.varVec.append(var)
self.blockSizes.append(blockSize)
self.blockingAvg = np.average(self.meanVec[-200:])
self.blockingVar = (np.average(self.varVec[-200:]))
self.blockingStd = np.sqrt(self.blockingVar)
------------------
 File "<ipython-input-20-f6d7a289d493>", line 3
 print(x)
 File "<ipython-input-6-2ff97f4bf03b>", line 2
 @timeFunction
 ^
SyntaxError: invalid syntax
IndentationError: unexpected indent
SyntaxError: invalid syntax (<ipython-input-20-f6d7a289d493>, line 3)
IndentationError: unexpected indent (<ipython-input-6-2ff97f4bf03b>, line 2)
doc/src/LectureNotes/_build/html/search.html
+6 -71
View File
@@ -93,99 +93,34 @@
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers">
2. Random Numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers-better-name-pseudo-random-numbers">
3. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng">
4. Random number generator RNG
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-periodic-outputs">
5. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-its-period">
6. Random number generator RNG and its period
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-other-examples">
7. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id9">
8. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0">
9. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id10">
10. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id11">
11. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id12">
12. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0-code">
13. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#which-rng-should-i-use">
14. Which RNG should I use?
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
15. Getting started, our first data and Machine Learning encounters
2. Getting started, our first data and Machine Learning encounters
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
16. Linear Regression and more Advanced Regression Analysis
3. Linear Regression and more Advanced Regression Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
17. Logistic Regression
4. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
18. Neural networks, from the simple perceptron to deep learning
5. Neural networks, from the simple perceptron to deep learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
19. Support Vector Machines, overarching aims
6. Support Vector Machines, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
20. Dimensionality Reduction
7. Dimensionality Reduction
</a>
</li>
</ul>
doc/src/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/src/LectureNotes/_build/jupyter_execute/chapter2.ipynb
+56 -46
View File
File diff suppressed because one or more lines are too long
doc/src/LectureNotes/_build/jupyter_execute/chapter2.py
+13 -13
View File
@@ -956,7 +956,7 @@ print(c)
Eigvals, Eigvecs = np.linalg.eig(c)
print(Eigvals)
# Random Numbers
## Random Numbers
Uniform deviates are just random numbers that lie within a specified range
(typically 0 to 1), with any one number in the range just as likely as any other. They
@@ -973,7 +973,7 @@ or Monte Carlo computer work.
# Random Numbers, better name: pseudo random numbers
## Random Numbers, better name: pseudo random numbers
A disclaimer is however appropriate. It should be fairly obvious that
something as deterministic as a computer cannot generate purely random numbers.
@@ -993,7 +993,7 @@ numbers, hopefully abiding to the following criteria:
# Random number generator RNG
## Random number generator RNG
The most common random number generators are based on so-called
Linear congruential relations of the type
@@ -1017,7 +1017,7 @@ of the division $13/9$, namely $4$.
# Random number generator RNG and periodic outputs
## Random number generator RNG and periodic outputs
The problem with such generators is that their outputs are periodic;
they
@@ -1045,7 +1045,7 @@ just $2$.
# Random number generator RNG and its period
## Random number generator RNG and its period
Typical periods for the random generators provided in the program library
are of the order of $\sim 10^9$ or larger. Other random number generators which have
become increasingly popular are so-called shift-register generators.
@@ -1059,7 +1059,7 @@ $$
N_l=(aN_{l-i}+cN_{l-j})\mathrm{MOD}(M).
$$
# Random number generator RNG, other examples
## Random number generator RNG, other examples
Such a generator again produces a sequence of pseudorandom numbers
but this time with a period much larger than $M$.
It is also possible to construct more elaborate algorithms by including
@@ -1094,7 +1094,7 @@ which according to the authors has a period larger than $2^{94}$.
# Random number generator RNG, other examples
## Random number generator RNG, other examples
Instead of using modular addition, we could use the bitwise
exclusive-OR ($\oplus$) operation so that
@@ -1117,7 +1117,7 @@ it is given by $m\wedge n$.
# Random number generator RNG, RAN0
## Random number generator RNG, RAN0
We show here how the linear congruential algorithm can be implemented, namely
@@ -1154,7 +1154,7 @@ $q$ and $r$ are chosen so that $r < q$.
# Random number generator RNG, RAN0
## Random number generator RNG, RAN0
To see how this works we note first that
@@ -1176,7 +1176,7 @@ $[N_{i-1}/q]$ just yields a constant which is multiplied with $M$.
# Random number generator RNG, RAN0
## Random number generator RNG, RAN0
We can now rewrite Eq. ([14](#eq:rntrick1)) as
<!-- Equation labels as ordinary links -->
@@ -1214,7 +1214,7 @@ $$
\end{equation}
$$
# Random number generator RNG, RAN0
## Random number generator RNG, RAN0
The term $[N_{i-1}/q]r$ is always smaller or equal $N_{i-1}(r/q)$ and with $r < q$ we obtain always a
number smaller than $N_{i-1}$, which is smaller than $M$.
And since the number $N_{i-1}\mathrm{MOD} (q)$ is between zero and $q-1$ then
@@ -1232,7 +1232,7 @@ set to $0$.
# Random number generator RNG, RAN0 code
## Random number generator RNG, RAN0 code
/*
** The function
@@ -1498,7 +1498,7 @@ The program here computes the correlation function for one of the standard funct
} // end of main program
# Which RNG should I use?
## Which RNG should I use?
* C++ has a class called **random**. The [random class](http://www.cplusplus.com/reference/random/) contains a large selection of RNGs and is highly recommended. Some of these RNGs have very large periods making it thereby very safe to use these RNGs in case one is performing large calculations. In particular, the [Mersenne twister random number engine](http://www.cplusplus.com/reference/random/mersenne_twister_engine/) has a period of $2^{19937}$.
* Add RNGs in Python
doc/src/LectureNotes/_build/jupyter_execute/chapter2_178_0.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 5.0 KiB

After

Width:  |  Height:  |  Size: 5.1 KiB

doc/src/LectureNotes/_build/jupyter_execute/chapter2_184_1.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 9.6 KiB

After

Width:  |  Height:  |  Size: 10 KiB

doc/src/LectureNotes/chapter2.ipynb
+13 -13
View File
@@ -1669,7 +1669,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Random Numbers\n",
"## Random Numbers\n",
"\n",
"Uniform deviates are just random numbers that lie within a specified range\n",
"(typically 0 to 1), with any one number in the range just as likely as any other. They\n",
@@ -1686,7 +1686,7 @@
"\n",
"\n",
"\n",
"# Random Numbers, better name: pseudo random numbers\n",
"## Random Numbers, better name: pseudo random numbers\n",
"\n",
"A disclaimer is however appropriate. It should be fairly obvious that \n",
"something as deterministic as a computer cannot generate purely random numbers.\n",
@@ -1706,7 +1706,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG\n",
"## Random number generator RNG\n",
" The most common random number generators are based on so-called\n",
"Linear congruential relations of the type"
]
@@ -1750,7 +1750,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG and periodic outputs\n",
"## Random number generator RNG and periodic outputs\n",
"\n",
"The problem with such generators is that their outputs are periodic;\n",
"they \n",
@@ -1798,7 +1798,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG and its period\n",
"## Random number generator RNG and its period\n",
"Typical periods for the random generators provided in the program library \n",
"are of the order of $\\sim 10^9$ or larger. Other random number generators which have\n",
"become increasingly popular are so-called shift-register generators.\n",
@@ -1822,7 +1822,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Random number generator RNG, other examples\n",
"## Random number generator RNG, other examples\n",
"Such a generator again produces a sequence of pseudorandom numbers\n",
"but this time with a period much larger than $M$.\n",
"It is also possible to construct more elaborate algorithms by including\n",
@@ -1877,7 +1877,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, other examples\n",
"## Random number generator RNG, other examples\n",
"Instead of using modular addition, we could use the bitwise\n",
"exclusive-OR ($\\oplus$) operation so that"
]
@@ -1910,7 +1910,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, RAN0\n",
"## Random number generator RNG, RAN0\n",
"\n",
"We show here how the linear congruential algorithm can be implemented, namely"
]
@@ -1987,7 +1987,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, RAN0\n",
"## Random number generator RNG, RAN0\n",
"\n",
"To see how this works we note first that"
]
@@ -2019,7 +2019,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, RAN0\n",
"## Random number generator RNG, RAN0\n",
"We can now rewrite Eq. ([14](#eq:rntrick1)) as"
]
},
@@ -2087,7 +2087,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Random number generator RNG, RAN0\n",
"## Random number generator RNG, RAN0\n",
"The term $[N_{i-1}/q]r$ is always smaller or equal $N_{i-1}(r/q)$ and with $r < q$ we obtain always a \n",
"number smaller than $N_{i-1}$, which is smaller than $M$. \n",
"And since the number $N_{i-1}\\mathrm{MOD} (q)$ is between zero and $q-1$ then\n",
@@ -2105,7 +2105,7 @@
"\n",
"\n",
"\n",
"# Random number generator RNG, RAN0 code"
"## Random number generator RNG, RAN0 code"
]
},
{
@@ -2457,7 +2457,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Which RNG should I use?\n",
"## Which RNG should I use?\n",
"* C++ has a class called **random**. The [random class](http://www.cplusplus.com/reference/random/) contains a large selection of RNGs and is highly recommended. Some of these RNGs have very large periods making it thereby very safe to use these RNGs in case one is performing large calculations. In particular, the [Mersenne twister random number engine](http://www.cplusplus.com/reference/random/mersenne_twister_engine/) has a period of $2^{19937}$. \n",
"\n",
"* Add RNGs in Python\n",
doc/src/LectureNotes/gaussian.pdf
BIN
View File
Binary file not shown.