lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

typos in reg slides

Browse Source
This commit is contained in:
mhjensen
2020-09-11 11:19:35 +02:00
parent 243a68c942
commit a6f462e86f
128 changed files with 21520 additions and 21811 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Regression/html/Regression.html
+207 -225
View File
@@ -150,154 +150,153 @@ div { text-align: justify; text-justify: inter-word; }
('Fixing the singularity', 2, None, '___sec35'),
('Basic math of the SVD', 2, None, '___sec36'),
('The SVD, a Fantastic Algorithm', 2, None, '___sec37'),
('Another Example', 2, None, '___sec38'),
('Economy-size SVD', 2, None, '___sec39'),
('Economy-size SVD', 2, None, '___sec38'),
('Codes for the SVD', 2, None, '___sec39'),
('Mathematical Properties', 2, None, '___sec40'),
('Ridge and LASSO Regression', 2, None, '___sec41'),
('More on Ridge Regression', 2, None, '___sec42'),
('Interpreting the Ridge results', 2, None, '___sec43'),
('More interpretations', 2, None, '___sec44'),
('Codes for the SVD', 2, None, '___sec45'),
('A better understanding of regularization', 2, None, '___sec46'),
('A better understanding of regularization', 2, None, '___sec45'),
('Decomposing the OLS and Ridge expressions',
2,
None,
'___sec47'),
'___sec46'),
('Introducing the Covariance and Correlation functions',
2,
None,
'___sec48'),
'___sec47'),
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec49'),
('Covariance Matrix Examples', 2, None, '___sec50'),
('Correlation Matrix', 2, None, '___sec51'),
('Correlation Matrix with Pandas', 2, None, '___sec52'),
'___sec48'),
('Covariance Matrix Examples', 2, None, '___sec49'),
('Correlation Matrix', 2, None, '___sec50'),
('Correlation Matrix with Pandas', 2, None, '___sec51'),
('Correlation Matrix with Pandas and the Franke function',
2,
None,
'___sec53'),
'___sec52'),
('Rewriting the Covariance and/or Correlation Matrix',
2,
None,
'___sec54'),
('Linking with SVD', 2, None, '___sec55'),
('Where are we going?', 2, None, '___sec56'),
('Resampling methods', 2, None, '___sec57'),
'___sec53'),
('Linking with SVD', 2, None, '___sec54'),
('Where are we going?', 2, None, '___sec55'),
('Resampling methods', 2, None, '___sec56'),
('Resampling approaches can be computationally expensive',
2,
None,
'___sec58'),
('Why resampling methods ?', 2, None, '___sec59'),
('Statistical analysis', 2, None, '___sec60'),
('Statistics', 2, None, '___sec61'),
('Statistics, moments', 2, None, '___sec62'),
('Statistics, central moments', 2, None, '___sec63'),
('Statistics, covariance', 2, None, '___sec64'),
('Statistics, more covariance', 2, None, '___sec65'),
('Covariance example', 2, None, '___sec66'),
('Covariance in numpy', 2, None, '___sec67'),
('Statistics, independent variables', 2, None, '___sec68'),
('Statistics, more variance', 2, None, '___sec69'),
('Statistics and stochastic processes', 2, None, '___sec70'),
('Statistics and sample variables', 2, None, '___sec71'),
'___sec57'),
('Why resampling methods ?', 2, None, '___sec58'),
('Statistical analysis', 2, None, '___sec59'),
('Statistics', 2, None, '___sec60'),
('Statistics, moments', 2, None, '___sec61'),
('Statistics, central moments', 2, None, '___sec62'),
('Statistics, covariance', 2, None, '___sec63'),
('Statistics, more covariance', 2, None, '___sec64'),
('Covariance example', 2, None, '___sec65'),
('Covariance in numpy', 2, None, '___sec66'),
('Statistics, independent variables', 2, None, '___sec67'),
('Statistics, more variance', 2, None, '___sec68'),
('Statistics and stochastic processes', 2, None, '___sec69'),
('Statistics and sample variables', 2, None, '___sec70'),
('Statistics, sample variance and covariance',
2,
None,
'___sec72'),
('Statistics, law of large numbers', 2, None, '___sec73'),
('Statistics, more on sample error', 2, None, '___sec74'),
('Statistics', 2, None, '___sec75'),
('Statistics, central limit theorem', 2, None, '___sec76'),
('Statistics, more technicalities', 2, None, '___sec77'),
('Statistics', 2, None, '___sec78'),
('Statistics and sample variance', 2, None, '___sec79'),
('Statistics, uncorrelated results', 2, None, '___sec80'),
('Statistics, computations', 2, None, '___sec81'),
'___sec71'),
('Statistics, law of large numbers', 2, None, '___sec72'),
('Statistics, more on sample error', 2, None, '___sec73'),
('Statistics', 2, None, '___sec74'),
('Statistics, central limit theorem', 2, None, '___sec75'),
('Statistics, more technicalities', 2, None, '___sec76'),
('Statistics', 2, None, '___sec77'),
('Statistics and sample variance', 2, None, '___sec78'),
('Statistics, uncorrelated results', 2, None, '___sec79'),
('Statistics, computations', 2, None, '___sec80'),
('Statistics, more on computations of errors',
2,
None,
'___sec82'),
('Statistics, wrapping up 1', 2, None, '___sec83'),
('Statistics, final expression', 2, None, '___sec84'),
'___sec81'),
('Statistics, wrapping up 1', 2, None, '___sec82'),
('Statistics, final expression', 2, None, '___sec83'),
('Statistics, effective number of correlations',
2,
None,
'___sec85'),
'___sec84'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
None,
'___sec86'),
('Assumptions made', 2, None, '___sec87'),
('Expectation value and variance', 2, None, '___sec88'),
'___sec85'),
('Assumptions made', 2, None, '___sec86'),
('Expectation value and variance', 2, None, '___sec87'),
('Expectation value and variance for $\\boldsymbol{\\beta}$',
2,
None,
'___sec89'),
('Resampling methods', 2, None, '___sec90'),
'___sec88'),
('Resampling methods', 2, None, '___sec89'),
('Resampling methods: Jackknife and Bootstrap',
2,
None,
'___sec91'),
('Resampling methods: Jackknife', 2, None, '___sec92'),
('Jackknife code example', 2, None, '___sec93'),
('Resampling methods: Bootstrap', 2, None, '___sec94'),
('Resampling methods: Bootstrap background', 2, None, '___sec95'),
'___sec90'),
('Resampling methods: Jackknife', 2, None, '___sec91'),
('Jackknife code example', 2, None, '___sec92'),
('Resampling methods: Bootstrap', 2, None, '___sec93'),
('Resampling methods: Bootstrap background', 2, None, '___sec94'),
('Resampling methods: More Bootstrap background',
2,
None,
'___sec96'),
('Resampling methods: Bootstrap approach', 2, None, '___sec97'),
('Resampling methods: Bootstrap steps', 2, None, '___sec98'),
('Code example for the Bootstrap method', 2, None, '___sec99'),
('Various steps in cross-validation', 2, None, '___sec100'),
'___sec95'),
('Resampling methods: Bootstrap approach', 2, None, '___sec96'),
('Resampling methods: Bootstrap steps', 2, None, '___sec97'),
('Code example for the Bootstrap method', 2, None, '___sec98'),
('Various steps in cross-validation', 2, None, '___sec99'),
('How to set up the cross-validation for Ridge and/or Lasso',
2,
None,
'___sec101'),
('Cross-validation in brief', 2, None, '___sec102'),
'___sec100'),
('Cross-validation in brief', 2, None, '___sec101'),
('Code Example for Cross-validation and $k$-fold '
'Cross-validation',
2,
None,
'___sec103'),
('The bias-variance tradeoff', 2, None, '___sec104'),
('Example code for Bias-Variance tradeoff', 2, None, '___sec105'),
('Understanding what happens', 2, None, '___sec106'),
('Summing up', 2, None, '___sec107'),
'___sec102'),
('The bias-variance tradeoff', 2, None, '___sec103'),
('Example code for Bias-Variance tradeoff', 2, None, '___sec104'),
('Understanding what happens', 2, None, '___sec105'),
('Summing up', 2, None, '___sec106'),
("Another Example from Scikit-Learn's Repository",
2,
None,
'___sec108'),
'___sec107'),
('More examples on bootstrap and cross-validation and errors',
2,
None,
'___sec109'),
'___sec108'),
('The same example but now with cross-validation',
2,
None,
'___sec110'),
('Cross-validation with Ridge', 2, None, '___sec111'),
('The Ising model', 2, None, '___sec112'),
'___sec109'),
('Cross-validation with Ridge', 2, None, '___sec110'),
('The Ising model', 2, None, '___sec111'),
('Reformulating the problem to suit regression',
2,
None,
'___sec113'),
('Linear regression', 2, None, '___sec114'),
('Singular Value decomposition', 2, None, '___sec115'),
('The one-dimensional Ising model', 2, None, '___sec116'),
('Ridge regression', 2, None, '___sec117'),
('LASSO regression', 2, None, '___sec118'),
'___sec112'),
('Linear regression', 2, None, '___sec113'),
('Singular Value decomposition', 2, None, '___sec114'),
('The one-dimensional Ising model', 2, None, '___sec115'),
('Ridge regression', 2, None, '___sec116'),
('LASSO regression', 2, None, '___sec117'),
('Performance as function of the regularization parameter',
2,
None,
'___sec119'),
'___sec118'),
('Finding the optimal value of $\\lambda$',
2,
None,
'___sec120')]}
'___sec119')]}
end of tocinfo -->
<body>
@@ -1802,7 +1801,7 @@ however not the be case in general and a standard matrix inversion
algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.
<p>
There is however a way to partially circumvent this problem and also gain some insight about the ordinary least squares approach.
There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions.
<p>
This is given by the <b>Singular Value Decomposition</b> algorithm, perhaps
@@ -1964,26 +1963,9 @@ with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \).
The SVD exits always!
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec38">Another Example </h2>
<p>
Consider the following matrix which can be SVD decomposed as
$$
\boldsymbol{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{bmatrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
<p>
This is a \( 3\times 2 \) matrix which is decomposed in terms of a
\( 3\times 3 \) matrix \( \boldsymbol{U} \), and a \( 2\times 2 \) matrix \( \boldsymbol{V} \). It is easy to see
that \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal (how?).
<p>
And the SVD
The SVD
decomposition (singular values) gives eigenvalues
\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=2 \), the
\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=p \), the
eigenvalues (singular values) are zero.
<p>
@@ -2004,7 +1986,7 @@ The columns of \( \boldsymbol{U} \) are called the left singular vectors while t
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">Economy-size SVD </h2>
<h2 id="___sec38">Economy-size SVD </h2>
<p>
If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n
@@ -2029,6 +2011,57 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">Codes for the SVD </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #408080; font-style: italic"># SVD inversion</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">SVDinv</span>(A):
<span style="color: #BA2121; font-style: italic">&#39;&#39;&#39; Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).</span>
<span style="color: #BA2121; font-style: italic"> SVD is numerically more stable than the inversion algorithms provided by</span>
<span style="color: #BA2121; font-style: italic"> numpy and scipy.linalg at the cost of being slower.</span>
<span style="color: #BA2121; font-style: italic"> &#39;&#39;&#39;</span>
U, s, VT <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>svd(A)
<span style="color: #408080; font-style: italic"># print(&#39;test U&#39;)</span>
<span style="color: #408080; font-style: italic"># print( (np.transpose(U) @ U - U @np.transpose(U)))</span>
<span style="color: #408080; font-style: italic"># print(&#39;test VT&#39;)</span>
<span style="color: #408080; font-style: italic"># print( (np.transpose(VT) @ VT - VT @np.transpose(VT)))</span>
<span style="color: #008000">print</span>(U)
<span style="color: #008000">print</span>(s)
<span style="color: #008000">print</span>(VT)
D <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(U),<span style="color: #008000">len</span>(VT)))
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">0</span>,<span style="color: #008000">len</span>(VT)):
D[i,i]<span style="color: #666666">=</span>s[i]
UT <span style="color: #666666">=</span> np<span style="color: #666666">.</span>transpose(U); V <span style="color: #666666">=</span> np<span style="color: #666666">.</span>transpose(VT); invD <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(D)
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>matmul(V,np<span style="color: #666666">.</span>matmul(invD,UT))
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([ [<span style="color: #666666">1.0</span>, <span style="color: #666666">-1.0</span>, <span style="color: #666666">2.0</span>], [<span style="color: #666666">1.0</span>, <span style="color: #666666">0.0</span>, <span style="color: #666666">1.0</span>], [<span style="color: #666666">1.0</span>, <span style="color: #666666">2.0</span>, <span style="color: #666666">-1.0</span>], [<span style="color: #666666">1.0</span>, <span style="color: #666666">1.0</span>, <span style="color: #666666">0.0</span>] ])
<span style="color: #008000">print</span>(X)
A <span style="color: #666666">=</span> np<span style="color: #666666">.</span>transpose(X) <span style="color: #666666">@</span> X
<span style="color: #008000">print</span>(A)
<span style="color: #408080; font-style: italic"># Brute force inversion of super-collinear matrix</span>
<span style="color: #408080; font-style: italic">#B = np.linalg.inv(A)</span>
<span style="color: #408080; font-style: italic">#print(B)</span>
C <span style="color: #666666">=</span> SVDinv(A)
<span style="color: #008000">print</span>(C)
</pre></div>
<p>
The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first
column is the row-wise sum of the other two columns. The rank of a
matrix (the column rank) is the dimension of space spanned by the
column vectors. The rank of the matrix is the number of linearly
independent columns, in this case just \( 2 \). We see this from the
singular values when running the above code. Running the standard
inversion algorithm for matrix inversion with \( \boldsymbol{X}^T\boldsymbol{X} \) results
in the program terminating due to a singular matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">Mathematical Properties </h2>
<p>
@@ -2256,61 +2289,10 @@ We will come back to more interpreations after we have gone through some of the
For more discussions of Ridge and Lasso regression, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.
Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et al's article</a> is also recommended.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec45">Codes for the SVD </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #408080; font-style: italic"># SVD inversion</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">SVDinv</span>(A):
<span style="color: #BA2121; font-style: italic">&#39;&#39;&#39; Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).</span>
<span style="color: #BA2121; font-style: italic"> SVD is numerically more stable than the inversion algorithms provided by</span>
<span style="color: #BA2121; font-style: italic"> numpy and scipy.linalg at the cost of being slower.</span>
<span style="color: #BA2121; font-style: italic"> &#39;&#39;&#39;</span>
U, s, VT <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>svd(A)
<span style="color: #408080; font-style: italic"># print(&#39;test U&#39;)</span>
<span style="color: #408080; font-style: italic"># print( (np.transpose(U) @ U - U @np.transpose(U)))</span>
<span style="color: #408080; font-style: italic"># print(&#39;test VT&#39;)</span>
<span style="color: #408080; font-style: italic"># print( (np.transpose(VT) @ VT - VT @np.transpose(VT)))</span>
<span style="color: #008000">print</span>(U)
<span style="color: #008000">print</span>(s)
<span style="color: #008000">print</span>(VT)
D <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(U),<span style="color: #008000">len</span>(VT)))
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">0</span>,<span style="color: #008000">len</span>(VT)):
D[i,i]<span style="color: #666666">=</span>s[i]
UT <span style="color: #666666">=</span> np<span style="color: #666666">.</span>transpose(U); V <span style="color: #666666">=</span> np<span style="color: #666666">.</span>transpose(VT); invD <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(D)
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>matmul(V,np<span style="color: #666666">.</span>matmul(invD,UT))
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([ [<span style="color: #666666">1.0</span>, <span style="color: #666666">-1.0</span>, <span style="color: #666666">2.0</span>], [<span style="color: #666666">1.0</span>, <span style="color: #666666">0.0</span>, <span style="color: #666666">1.0</span>], [<span style="color: #666666">1.0</span>, <span style="color: #666666">2.0</span>, <span style="color: #666666">-1.0</span>], [<span style="color: #666666">1.0</span>, <span style="color: #666666">1.0</span>, <span style="color: #666666">0.0</span>] ])
<span style="color: #008000">print</span>(X)
A <span style="color: #666666">=</span> np<span style="color: #666666">.</span>transpose(X) <span style="color: #666666">@</span> X
<span style="color: #008000">print</span>(A)
<span style="color: #408080; font-style: italic"># Brute force inversion of super-collinear matrix</span>
<span style="color: #408080; font-style: italic">#B = np.linalg.inv(A)</span>
<span style="color: #408080; font-style: italic">#print(B)</span>
C <span style="color: #666666">=</span> SVDinv(A)
<span style="color: #008000">print</span>(C)
</pre></div>
<p>
The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first
column is the row-wise sum of the other two columns. The rank of a
matrix (the column rank) is the dimension of space spanned by the
column vectors. The rank of the matrix is the number of linearly
independent columns, in this case just \( 2 \). We see this from the
singular values when running the above code. Running the standard
inversion algorithm for matrix inversion with \( \boldsymbol{X}^T\boldsymbol{X} \) results
in the program terminating due to a singular matrix.
<p>
<!-- !split -->
<h2 id="___sec46">A better understanding of regularization </h2>
<h2 id="___sec45">A better understanding of regularization </h2>
<p>
The parameter \( \lambda \) that we have introduced in the Ridge (and
@@ -2329,7 +2311,7 @@ affected by changing the parameter \( \lambda \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec47">Decomposing the OLS and Ridge expressions </h2>
<h2 id="___sec46">Decomposing the OLS and Ridge expressions </h2>
<p>
We have our design matrix
@@ -2349,7 +2331,7 @@ The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonorm
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">Introducing the Covariance and Correlation functions </h2>
<h2 id="___sec47">Introducing the Covariance and Correlation functions </h2>
<p>
Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about
@@ -2410,7 +2392,7 @@ In the above example this is the function we constructed using <b>pandas</b>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec49">Correlation Function and Design/Feature Matrix </h2>
<h2 id="___sec48">Correlation Function and Design/Feature Matrix </h2>
<p>
In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
@@ -2471,7 +2453,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec50">Covariance Matrix Examples </h2>
<h2 id="___sec49">Covariance Matrix Examples </h2>
<p>
The Numpy function <b>np.cov</b> calculates the covariance elements using
@@ -2514,7 +2496,7 @@ C <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Correlation Matrix </h2>
<h2 id="___sec50">Correlation Matrix </h2>
<p>
The previous example can be converted into the correlation matrix by
@@ -2559,7 +2541,7 @@ The above procedure with <b>numpy</b> can be made more compact if we use <b>pand
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec52">Correlation Matrix with Pandas </h2>
<h2 id="___sec51">Correlation Matrix with Pandas </h2>
<p>
We whow here how we can set up the correlation matrix using <b>pandas</b>, as done in this simple code
@@ -2586,7 +2568,7 @@ We expand this model to the Franke function discussed above.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec53">Correlation Matrix with Pandas and the Franke function </h2>
<h2 id="___sec52">Correlation Matrix with Pandas and the Franke function </h2>
<p>
@@ -2649,7 +2631,7 @@ matrix without these elements.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec54">Rewriting the Covariance and/or Correlation Matrix </h2>
<h2 id="___sec53">Rewriting the Covariance and/or Correlation Matrix </h2>
<p>
We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as
@@ -2692,7 +2674,7 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec55">Linking with SVD </h2>
<h2 id="___sec54">Linking with SVD </h2>
<p>
See lecture september 11. More text to be added here soon.
@@ -2700,7 +2682,7 @@ See lecture september 11. More text to be added here soon.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec56">Where are we going? </h2>
<h2 id="___sec55">Where are we going? </h2>
<p>
Before we proceed, we need to rethink what we have been doing. In our
@@ -2717,7 +2699,7 @@ This will allow us to link the standard linear algebra methods we have discussed
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec57">Resampling methods </h2>
<h2 id="___sec56">Resampling methods </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2750,7 +2732,7 @@ cross-validation and the bootstrap method.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec58">Resampling approaches can be computationally expensive </h2>
<h2 id="___sec57">Resampling approaches can be computationally expensive </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2779,7 +2761,7 @@ bootstrap is widely used.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec59">Why resampling methods ? </h2>
<h2 id="___sec58">Why resampling methods ? </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b>Statistical analysis.</b>
<p>
@@ -2795,7 +2777,7 @@ bootstrap is widely used.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec60">Statistical analysis </h2>
<h2 id="___sec59">Statistical analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2817,7 +2799,7 @@ bootstrap is widely used.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec61">Statistics </h2>
<h2 id="___sec60">Statistics </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2848,7 +2830,7 @@ selection of a large set of these numbers reproduces this PDF.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec62">Statistics, moments </h2>
<h2 id="___sec61">Statistics, moments </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2871,7 +2853,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec63">Statistics, central moments </h2>
<h2 id="___sec62">Statistics, central moments </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2909,7 +2891,7 @@ qualitatively as the <em>spread</em> of \( p \) around its mean.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec64">Statistics, covariance </h2>
<h2 id="___sec63">Statistics, covariance </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2940,7 +2922,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec65">Statistics, more covariance </h2>
<h2 id="___sec64">Statistics, more covariance </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2972,7 +2954,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec66">Covariance example </h2>
<h2 id="___sec65">Covariance example </h2>
<p>
Suppose we have defined three vectors \( \boldsymbol{x}, \boldsymbol{y}, \boldsymbol{z} \) with
@@ -3021,7 +3003,7 @@ function.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec67">Covariance in numpy </h2>
<h2 id="___sec66">Covariance in numpy </h2>
<p>
@@ -3060,7 +3042,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec68">Statistics, independent variables </h2>
<h2 id="___sec67">Statistics, independent variables </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3086,7 +3068,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec69">Statistics, more variance </h2>
<h2 id="___sec68">Statistics, more variance </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3118,7 +3100,7 @@ value of a set of measurements.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec70">Statistics and stochastic processes </h2>
<h2 id="___sec69">Statistics and stochastic processes </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3146,7 +3128,7 @@ interested in finding the few lowest moments, like the mean
<p>
<!-- !split -->
<h2 id="___sec71">Statistics and sample variables </h2>
<h2 id="___sec70">Statistics and sample variables </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3172,7 +3154,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec72">Statistics, sample variance and covariance </h2>
<h2 id="___sec71">Statistics, sample variance and covariance </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3193,7 +3175,7 @@ and covariance \( \mathrm{cov}(X,Y) \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec73">Statistics, law of large numbers </h2>
<h2 id="___sec72">Statistics, law of large numbers </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3225,7 +3207,7 @@ true PDFs behind, which we usually do not have.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec74">Statistics, more on sample error </h2>
<h2 id="___sec73">Statistics, more on sample error </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3247,7 +3229,7 @@ means.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec75">Statistics </h2>
<h2 id="___sec74">Statistics </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3268,7 +3250,7 @@ And in particular we are interested in its variance \( \mathrm{var}(\overline X_
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec76">Statistics, central limit theorem </h2>
<h2 id="___sec75">Statistics, central limit theorem </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3293,7 +3275,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec77">Statistics, more technicalities </h2>
<h2 id="___sec76">Statistics, more technicalities </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3322,7 +3304,7 @@ estimate of the PDF of each of the \( X_i \), estimating all properties of
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec78">Statistics </h2>
<h2 id="___sec77">Statistics </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3349,7 +3331,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec79">Statistics and sample variance </h2>
<h2 id="___sec78">Statistics and sample variance </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3388,7 +3370,7 @@ measurements in the sample.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec80">Statistics, uncorrelated results </h2>
<h2 id="___sec79">Statistics, uncorrelated results </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3423,7 +3405,7 @@ cannot overlook the always present correlations.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec81">Statistics, computations </h2>
<h2 id="___sec80">Statistics, computations </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3452,7 +3434,7 @@ measurements. For uncorrelated measurements this second term is zero.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec82">Statistics, more on computations of errors </h2>
<h2 id="___sec81">Statistics, more on computations of errors </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3474,7 +3456,7 @@ have to be stored throughout the experiment.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec83">Statistics, wrapping up 1 </h2>
<h2 id="___sec82">Statistics, wrapping up 1 </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3507,7 +3489,7 @@ starting always at \( 1 \) for \( d=0 \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec84">Statistics, final expression </h2>
<h2 id="___sec83">Statistics, final expression </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3540,7 +3522,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec85">Statistics, effective number of correlations </h2>
<h2 id="___sec84">Statistics, effective number of correlations </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3565,7 +3547,7 @@ measurements is very large.
<p>
<!-- !split -->
<h2 id="___sec86">Linking the regression analysis with a statistical interpretation </h2>
<h2 id="___sec85">Linking the regression analysis with a statistical interpretation </h2>
<p>
Finally, we are going to discuss several statistical properties which can be obtained in terms of analytical expressions.
@@ -3603,7 +3585,7 @@ row number \( i \) and perform a sum over all values \( p \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec87">Assumptions made </h2>
<h2 id="___sec86">Assumptions made </h2>
<p>
The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
@@ -3623,7 +3605,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec88">Expectation value and variance </h2>
<h2 id="___sec87">Expectation value and variance </h2>
<p>
We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
@@ -3658,7 +3640,7 @@ mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (n
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec89">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
<h2 id="___sec88">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
<p>
With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value
@@ -3743,7 +3725,7 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec90">Resampling methods </h2>
<h2 id="___sec89">Resampling methods </h2>
<p>
With all these analytical equations for both the OLS and Ridge
@@ -3772,7 +3754,7 @@ training error reaches a saturation.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec91">Resampling methods: Jackknife and Bootstrap </h2>
<h2 id="___sec90">Resampling methods: Jackknife and Bootstrap </h2>
<p>
Two famous
@@ -3795,7 +3777,7 @@ need for bootstrapping.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec92">Resampling methods: Jackknife </h2>
<h2 id="___sec91">Resampling methods: Jackknife </h2>
<p>
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
@@ -3814,7 +3796,7 @@ number \( i \) is left out. Using this notation, define
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec93">Jackknife code example </h2>
<h2 id="___sec92">Jackknife code example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -3850,7 +3832,7 @@ t <span style="color: #666666">=</span> jackknife(x, stat)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec94">Resampling methods: Bootstrap </h2>
<h2 id="___sec93">Resampling methods: Bootstrap </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -3871,7 +3853,7 @@ advantages:
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec95">Resampling methods: Bootstrap background </h2>
<h2 id="___sec94">Resampling methods: Bootstrap background </h2>
<p>
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
@@ -3887,7 +3869,7 @@ estimators.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec96">Resampling methods: More Bootstrap background </h2>
<h2 id="___sec95">Resampling methods: More Bootstrap background </h2>
<p>
In the case that \( \widehat{\theta} \) has
@@ -3909,7 +3891,7 @@ idea is to use the relative frequency of \( \widehat{\theta}^* \)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec97">Resampling methods: Bootstrap approach </h2>
<h2 id="___sec96">Resampling methods: Bootstrap approach </h2>
<p>
But
@@ -3930,7 +3912,7 @@ frequency of the observation \( X_i \), just draw the values
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec98">Resampling methods: Bootstrap steps </h2>
<h2 id="___sec97">Resampling methods: Bootstrap steps </h2>
<p>
The independent bootstrap works like this:
@@ -3955,7 +3937,7 @@ example, if you are interested in estimating the variance of \( \widehat
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec99">Code example for the Bootstrap method </h2>
<h2 id="___sec98">Code example for the Bootstrap method </h2>
<p>
The following code starts with a Gaussian distribution with mean value
@@ -4019,7 +4001,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split -->
<h2 id="___sec100">Various steps in cross-validation </h2>
<h2 id="___sec99">Various steps in cross-validation </h2>
<p>
When the repetitive splitting of the data set is done randomly,
@@ -4040,7 +4022,7 @@ cross-validation (LOOCV).
<p>
<!-- !split -->
<h2 id="___sec101">How to set up the cross-validation for Ridge and/or Lasso </h2>
<h2 id="___sec100">How to set up the cross-validation for Ridge and/or Lasso </h2>
<ul>
<li> Define a range of interest for the penalty parameter.</li>
@@ -4072,7 +4054,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec102">Cross-validation in brief </h2>
<h2 id="___sec101">Cross-validation in brief </h2>
<p>
For the various values of \( k \)
@@ -4094,7 +4076,7 @@ For the various values of \( k \)
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec103">Code Example for Cross-validation and \( k \)-fold Cross-validation </h2>
<h2 id="___sec102">Code Example for Cross-validation and \( k \)-fold Cross-validation </h2>
<p>
The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial.
@@ -4194,7 +4176,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec104">The bias-variance tradeoff </h2>
<h2 id="___sec103">The bias-variance tradeoff </h2>
<p>
We will discuss the bias-variance tradeoff in the context of
@@ -4260,7 +4242,7 @@ that is the rewriting in terms of the so-called bias, the variance of the model
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec105">Example code for Bias-Variance tradeoff </h2>
<h2 id="___sec104">Example code for Bias-Variance tradeoff </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -4322,7 +4304,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec106">Understanding what happens </h2>
<h2 id="___sec105">Understanding what happens </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -4376,7 +4358,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split -->
<h2 id="___sec107">Summing up </h2>
<h2 id="___sec106">Summing up </h2>
<p>
The bias-variance tradeoff summarizes the fundamental tension in
@@ -4413,7 +4395,7 @@ You may also find this recent <a href="https://www.pnas.org/content/116/32/15849
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec108">Another Example from Scikit-Learn's Repository </h2>
<h2 id="___sec107">Another Example from Scikit-Learn's Repository </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -4492,7 +4474,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec109">More examples on bootstrap and cross-validation and errors </h2>
<h2 id="___sec108">More examples on bootstrap and cross-validation and errors </h2>
<p>
@@ -4579,7 +4561,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split -->
<h2 id="___sec110">The same example but now with cross-validation </h2>
<h2 id="___sec109">The same example but now with cross-validation </h2>
<p>
@@ -4655,7 +4637,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec111">Cross-validation with Ridge </h2>
<h2 id="___sec110">Cross-validation with Ridge </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -4698,7 +4680,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec112">The Ising model </h2>
<h2 id="___sec111">The Ising model </h2>
<p>
The one-dimensional Ising model with nearest neighbor interaction, no
@@ -4755,7 +4737,7 @@ the coupling constant to achieve this.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec113">Reformulating the problem to suit regression </h2>
<h2 id="___sec112">Reformulating the problem to suit regression </h2>
<p>
A more general form for the one-dimensional Ising model is
@@ -4805,7 +4787,7 @@ X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_tes
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec114">Linear regression </h2>
<h2 id="___sec113">Linear regression </h2>
<p>
In the ordinary least squares method we choose the cost function
@@ -4853,7 +4835,7 @@ beta <span style="color: #666666">=</span> ols_inv(X_train_own, y_train)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec115">Singular Value decomposition </h2>
<h2 id="___sec114">Singular Value decomposition </h2>
<p>
Doing the inversion directly turns out to be a bad idea since the matrix
@@ -4940,7 +4922,7 @@ In this case our matrix inversion was actually possible. The obvious question no
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec116">The one-dimensional Ising model </h2>
<h2 id="___sec115">The one-dimensional Ising model </h2>
<p>
Let us bring back the Ising model again, but now with an additional
@@ -5074,7 +5056,7 @@ The results perfectly with our previous discussion where we used our own code.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec117">Ridge regression </h2>
<h2 id="___sec116">Ridge regression </h2>
<p>
Having explored the ordinary least squares we move on to ridge
@@ -5109,7 +5091,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec118">LASSO regression </h2>
<h2 id="___sec117">LASSO regression </h2>
<p>
In the <b>Least Absolute Shrinkage and Selection Operator</b> (LASSO)-method we get a third cost function.
@@ -5147,7 +5129,7 @@ constant as opposed to ridge and OLS. We get a sparse solution with
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec119">Performance as function of the regularization parameter </h2>
<h2 id="___sec118">Performance as function of the regularization parameter </h2>
<p>
We see how the different models perform for a different set of values for \( \lambda \).
@@ -5201,7 +5183,7 @@ much. Ridge is more stable over a larger range of values for
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec120">Finding the optimal value of \( \lambda \) </h2>
<h2 id="___sec119">Finding the optimal value of \( \lambda \) </h2>
<p>
To determine which value of \( \lambda \) is best we plot the accuracy of