lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated regression analysis, sad to come

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2017-10-17 23:35:46 +02:00
parent 8df889b53d
commit 5cd9a2f279
18 changed files with 1774 additions and 177 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Regression/html/._Regression-bs000.html
+19 -4
View File
@@ -50,7 +50,16 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4')]}
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
@@ -65,8 +74,8 @@ MathJax.Hub.Config({
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
@@ -93,6 +102,9 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
@@ -127,7 +139,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 14, 2017</h4></center> <!-- date -->
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
<br>
<p>
@@ -146,6 +158,9 @@ MathJax.Hub.Config({
<li><a href="._Regression-bs003.html">4</a></li>
<li><a href="._Regression-bs004.html">5</a></li>
<li><a href="._Regression-bs005.html">6</a></li>
<li><a href="._Regression-bs006.html">7</a></li>
<li><a href="._Regression-bs007.html">8</a></li>
<li><a href="._Regression-bs008.html">9</a></li>
<li><a href="._Regression-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/Regression/html/._Regression-bs001.html
+18 -3
View File
@@ -50,7 +50,16 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4')]}
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
@@ -65,8 +74,8 @@ MathJax.Hub.Config({
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
@@ -93,6 +102,9 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
@@ -142,6 +154,9 @@ A regression model aims at finding a likelihood function \( p(y\vert \hat{x}) \)
<li><a href="._Regression-bs003.html">4</a></li>
<li><a href="._Regression-bs004.html">5</a></li>
<li><a href="._Regression-bs005.html">6</a></li>
<li><a href="._Regression-bs006.html">7</a></li>
<li><a href="._Regression-bs007.html">8</a></li>
<li><a href="._Regression-bs008.html">9</a></li>
<li><a href="._Regression-bs002.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/Regression/html/._Regression-bs002.html
+18 -3
View File
@@ -50,7 +50,16 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4')]}
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
@@ -65,8 +74,8 @@ MathJax.Hub.Config({
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
@@ -93,6 +102,9 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
@@ -136,6 +148,9 @@ where \( \epsilon_i \) is the error in our approximation.
<li><a href="._Regression-bs003.html">4</a></li>
<li><a href="._Regression-bs004.html">5</a></li>
<li><a href="._Regression-bs005.html">6</a></li>
<li><a href="._Regression-bs006.html">7</a></li>
<li><a href="._Regression-bs007.html">8</a></li>
<li><a href="._Regression-bs008.html">9</a></li>
<li><a href="._Regression-bs003.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/Regression/html/._Regression-bs003.html
+209
View File
@@ -0,0 +1,209 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
<!-- Bootstrap style: bootstrap -->
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
('General linear models', 2, None, '___sec1'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec2'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0003"></a>
<!-- !split -->
<h2 id="___sec2" class="anchor">Rewriting the fitting procedure as a linear algebra problem </h2>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
For every set of values \( y_i,x_i \) we have thus the corresponding set of equations
$$
\begin{align*}
y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\
y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\
y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
\dots & \dots \\
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
Defining the vectors
$$
\hat{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T,
$$
$$
\hat{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T,
$$
$$
\hat{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T,
$$
and the matrix
$$
\hat{X}=
\begin{bmatrix}
1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\
1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\
1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\
\end{bmatrix}
$$
we can rewrite our equations as
$$
\hat{y} = \hat{X}\hat{\beta}+\hat{\epsilon}.
$$
</div>
</div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._Regression-bs002.html">&laquo;</a></li>
<li><a href="._Regression-bs000.html">1</a></li>
<li><a href="._Regression-bs001.html">2</a></li>
<li><a href="._Regression-bs002.html">3</a></li>
<li class="active"><a href="._Regression-bs003.html">4</a></li>
<li><a href="._Regression-bs004.html">5</a></li>
<li><a href="._Regression-bs005.html">6</a></li>
<li><a href="._Regression-bs006.html">7</a></li>
<li><a href="._Regression-bs007.html">8</a></li>
<li><a href="._Regression-bs008.html">9</a></li>
<li><a href="._Regression-bs004.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/Regression/html/._Regression-bs004.html
+201
View File
@@ -0,0 +1,201 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
<!-- Bootstrap style: bootstrap -->
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
('General linear models', 2, None, '___sec1'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec2'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0004"></a>
<!-- !split -->
<h2 id="___sec3" class="anchor">Generalizing the fitting procedure as a linear algebra problem </h2>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
We are obviously not limited to the above polynomial. We could replace the various powers of \( x \) with elements of Fourier series, that is, instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions.
For every set of values \( y_i,x_i \) we can then generalize the equations to
$$
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
\dots & \dots \\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
We redefine in turn the matrix \( \hat{X} \) as
$$
\hat{X}=
\begin{bmatrix}
x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\
x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\
x_{20}& x_{21}^1 &x_{22}2& \dots & \dots &x_{2,n-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
x_{n-1,00}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\
\end{bmatrix}
$$
and without loss of generality we rewrite again our equations as
$$
\hat{y} = \hat{X}\hat{\beta}+\hat{\epsilon}.
$$
The left-hand side of this equation forms know. Our error vector \( \hat{\epsilon} \) and the parameter vector \( \hat{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values?
</div>
</div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._Regression-bs003.html">&laquo;</a></li>
<li><a href="._Regression-bs000.html">1</a></li>
<li><a href="._Regression-bs001.html">2</a></li>
<li><a href="._Regression-bs002.html">3</a></li>
<li><a href="._Regression-bs003.html">4</a></li>
<li class="active"><a href="._Regression-bs004.html">5</a></li>
<li><a href="._Regression-bs005.html">6</a></li>
<li><a href="._Regression-bs006.html">7</a></li>
<li><a href="._Regression-bs007.html">8</a></li>
<li><a href="._Regression-bs008.html">9</a></li>
<li><a href="._Regression-bs005.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/Regression/html/._Regression-bs005.html
+196
View File
@@ -0,0 +1,196 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
<!-- Bootstrap style: bootstrap -->
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
('General linear models', 2, None, '___sec1'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec2'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0005"></a>
<!-- !split -->
<h2 id="___sec4" class="anchor">Optimizing our parameters </h2>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
We have defined the matrix \( \hat{X} \)
$$
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
\dots & \dots \\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
$$
\hat{\tilde{y}}= \hat{X}\hat{\beta},
$$
and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parametrized values \( \tilde{y}_i \), namely
$$
Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
$$
or using the matrix \( \hat{X} \) as
$$
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
$$
</div>
</div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._Regression-bs004.html">&laquo;</a></li>
<li><a href="._Regression-bs000.html">1</a></li>
<li><a href="._Regression-bs001.html">2</a></li>
<li><a href="._Regression-bs002.html">3</a></li>
<li><a href="._Regression-bs003.html">4</a></li>
<li><a href="._Regression-bs004.html">5</a></li>
<li class="active"><a href="._Regression-bs005.html">6</a></li>
<li><a href="._Regression-bs006.html">7</a></li>
<li><a href="._Regression-bs007.html">8</a></li>
<li><a href="._Regression-bs008.html">9</a></li>
<li><a href="._Regression-bs006.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/Regression/html/._Regression-bs006.html
+198
View File
@@ -0,0 +1,198 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
<!-- Bootstrap style: bootstrap -->
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
('General linear models', 2, None, '___sec1'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec2'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0006"></a>
<!-- !split -->
<h2 id="___sec5" class="anchor">Interpretations and optimizing our parameters </h2>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
The function
$$
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
$$
can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
$$
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
$$
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
<p>
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
$$
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
which results in
$$
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
$$
or in a matrix-vector form as
$$
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right).
$$
<p>
</div>
</div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._Regression-bs005.html">&laquo;</a></li>
<li><a href="._Regression-bs000.html">1</a></li>
<li><a href="._Regression-bs001.html">2</a></li>
<li><a href="._Regression-bs002.html">3</a></li>
<li><a href="._Regression-bs003.html">4</a></li>
<li><a href="._Regression-bs004.html">5</a></li>
<li><a href="._Regression-bs005.html">6</a></li>
<li class="active"><a href="._Regression-bs006.html">7</a></li>
<li><a href="._Regression-bs007.html">8</a></li>
<li><a href="._Regression-bs008.html">9</a></li>
<li><a href="._Regression-bs007.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/Regression/html/._Regression-bs007.html
+202
View File
@@ -0,0 +1,202 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
<!-- Bootstrap style: bootstrap -->
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
('General linear models', 2, None, '___sec1'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec2'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0007"></a>
<!-- !split -->
<h2 id="___sec6" class="anchor">Interpretations and optimizing our parameters </h2>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
We can rewrite
$$
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),
$$
as
$$
\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},
$$
and if the matrix \( \hat{X}^T\hat{X} \) is invertible we have the solution
$$
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
$$
The residuals \( \hat{\epsilon} \) are in turn given by
$$
\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
$$
and with
$$
\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
$$
we have
$$
\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
$$
meaning that the solution for \( \hat{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.
<p>
</div>
</div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._Regression-bs006.html">&laquo;</a></li>
<li><a href="._Regression-bs000.html">1</a></li>
<li><a href="._Regression-bs001.html">2</a></li>
<li><a href="._Regression-bs002.html">3</a></li>
<li><a href="._Regression-bs003.html">4</a></li>
<li><a href="._Regression-bs004.html">5</a></li>
<li><a href="._Regression-bs005.html">6</a></li>
<li><a href="._Regression-bs006.html">7</a></li>
<li class="active"><a href="._Regression-bs007.html">8</a></li>
<li><a href="._Regression-bs008.html">9</a></li>
<li><a href="._Regression-bs008.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/Regression/html/._Regression-bs008.html
+170
View File
@@ -0,0 +1,170 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
<!-- Bootstrap style: bootstrap -->
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
('General linear models', 2, None, '___sec1'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec2'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0008"></a>
<!-- !split -->
<h2 id="___sec7" class="anchor">The singular value decompostion </h2>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
Here we derive the equations for the SVD.
</div>
</div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._Regression-bs007.html">&laquo;</a></li>
<li><a href="._Regression-bs000.html">1</a></li>
<li><a href="._Regression-bs001.html">2</a></li>
<li><a href="._Regression-bs002.html">3</a></li>
<li><a href="._Regression-bs003.html">4</a></li>
<li><a href="._Regression-bs004.html">5</a></li>
<li><a href="._Regression-bs005.html">6</a></li>
<li><a href="._Regression-bs006.html">7</a></li>
<li><a href="._Regression-bs007.html">8</a></li>
<li class="active"><a href="._Regression-bs008.html">9</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/Regression/html/Regression-bs.html
+19 -4
View File
@@ -50,7 +50,16 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4')]}
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
@@ -65,8 +74,8 @@ MathJax.Hub.Config({
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
@@ -93,6 +102,9 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
</ul>
</li>
@@ -127,7 +139,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 14, 2017</h4></center> <!-- date -->
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
<br>
<p>
@@ -146,6 +158,9 @@ MathJax.Hub.Config({
<li><a href="._Regression-bs003.html">4</a></li>
<li><a href="._Regression-bs004.html">5</a></li>
<li><a href="._Regression-bs005.html">6</a></li>
<li><a href="._Regression-bs006.html">7</a></li>
<li><a href="._Regression-bs007.html">8</a></li>
<li><a href="._Regression-bs008.html">9</a></li>
<li><a href="._Regression-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/Regression/html/Regression-reveal.html
+163 -51
View File
@@ -1,4 +1,3 @@
\
<!DOCTYPE html>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
@@ -117,8 +116,8 @@ MathJax.Hub.Config({
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
@@ -148,7 +147,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Oct 14, 2017</h4></center> <!-- date -->
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
<br>
<p>
@@ -211,19 +210,13 @@ where \( \epsilon_i \) is the error in our approximation.
For every set of values \( y_i,x_i \) we have thus the corresponding set of equations
<p>&nbsp;<br>
$$
\begin{align}
y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0
\tag{1}\\
y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1
\tag{2}\\
y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2
\tag{3}\\
\dots & \dots
\tag{4}\\
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.
\tag{5}\\
\tag{6}
\end{align}
\begin{align*}
y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\
y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\
y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
\dots & \dots \\
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
<p>&nbsp;<br>
@@ -279,23 +272,15 @@ We are obviously not limited to the above polynomial. We could replace the vario
For every set of values \( y_i,x_i \) we can then generalize the equations to
<p>&nbsp;<br>
$$
\begin{align}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0
\tag{7}\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1
\tag{8}\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1
\tag{9}\\
\dots & \dots
\tag{10}\\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1
\tag{11}\\
\dots & \dots
\tag{12}\\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.
\tag{13}\\
\tag{14}
\end{align}
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
\dots & \dots \\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
<p>&nbsp;<br>
@@ -329,27 +314,154 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
<h2 id="___sec4">Optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
We have defined the matrix \( \hat{X} \)
<p>&nbsp;<br>
$$
\begin{align}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0
\tag{15}\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1
\tag{16}\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1
\tag{17}\\
\dots & \dots
\tag{18}\\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1
\tag{19}\\
\dots & \dots
\tag{20}\\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.
\tag{21}\\
\tag{22}
\end{align}
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
\dots & \dots \\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
<p>&nbsp;<br>
We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
<p>&nbsp;<br>
$$
\hat{\tilde{y}}= \hat{X}\hat{\beta},
$$
<p>&nbsp;<br>
and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parametrized values \( \tilde{y}_i \), namely
<p>&nbsp;<br>
$$
Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
$$
<p>&nbsp;<br>
or using the matrix \( \hat{X} \) as
<p>&nbsp;<br>
$$
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
$$
<p>&nbsp;<br>
</div>
</section>
<section>
<h2 id="___sec5">Interpretations and optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
The function
<p>&nbsp;<br>
$$
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
$$
<p>&nbsp;<br>
can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
<p>&nbsp;<br>
$$
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
$$
<p>&nbsp;<br>
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
<p>
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
<p>&nbsp;<br>
$$
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
<p>&nbsp;<br>
which results in
<p>&nbsp;<br>
$$
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
$$
<p>&nbsp;<br>
or in a matrix-vector form as
<p>&nbsp;<br>
$$
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right).
$$
<p>&nbsp;<br>
</div>
</section>
<section>
<h2 id="___sec6">Interpretations and optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
We can rewrite
<p>&nbsp;<br>
$$
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),
$$
<p>&nbsp;<br>
as
<p>&nbsp;<br>
$$
\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},
$$
<p>&nbsp;<br>
and if the matrix \( \hat{X}^T\hat{X} \) is invertible we have the solution
<p>&nbsp;<br>
$$
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
$$
<p>&nbsp;<br>
The residuals \( \hat{\epsilon} \) are in turn given by
<p>&nbsp;<br>
$$
\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
$$
<p>&nbsp;<br>
and with
<p>&nbsp;<br>
$$
\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
$$
<p>&nbsp;<br>
we have
<p>&nbsp;<br>
$$
\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
$$
<p>&nbsp;<br>
meaning that the solution for \( \hat{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.
</div>
</section>
<section>
<h2 id="___sec7">The singular value decompostion </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Here we derive the equations for the SVD.
</div>
</section>
doc/pub/Regression/html/Regression-solarized.html
+149 -51
View File
@@ -70,7 +70,16 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4')]}
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
@@ -85,8 +94,8 @@ MathJax.Hub.Config({
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
@@ -112,7 +121,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 14, 2017</h4></center> <!-- date -->
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -167,19 +176,13 @@ where \( \epsilon_i \) is the error in our approximation.
<p>
For every set of values \( y_i,x_i \) we have thus the corresponding set of equations
$$
\begin{align}
y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0
\label{_auto1}\\
y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1
\label{_auto2}\\
y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2
\label{_auto3}\\
\dots & \dots
\label{_auto4}\\
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.
\label{_auto5}\\
\label{_auto6}
\end{align}
\begin{align*}
y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\
y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\
y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
\dots & \dots \\
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
Defining the vectors
@@ -224,23 +227,15 @@ $$
We are obviously not limited to the above polynomial. We could replace the various powers of \( x \) with elements of Fourier series, that is, instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions.
For every set of values \( y_i,x_i \) we can then generalize the equations to
$$
\begin{align}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0
\label{_auto7}\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1
\label{_auto8}\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1
\label{_auto9}\\
\dots & \dots
\label{_auto10}\\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1
\label{_auto11}\\
\dots & \dots
\label{_auto12}\\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.
\label{_auto13}\\
\label{_auto14}
\end{align}
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
\dots & \dots \\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
We redefine in turn the matrix \( \hat{X} \) as
@@ -271,28 +266,131 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
We have defined the matrix \( \hat{X} \)
$$
\begin{align}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0
\label{_auto15}\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1
\label{_auto16}\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1
\label{_auto17}\\
\dots & \dots
\label{_auto18}\\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1
\label{_auto19}\\
\dots & \dots
\label{_auto20}\\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.
\label{_auto21}\\
\label{_auto22}
\end{align}
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
\dots & \dots \\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
$$
\hat{\tilde{y}}= \hat{X}\hat{\beta},
$$
and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parametrized values \( \tilde{y}_i \), namely
$$
Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
$$
or using the matrix \( \hat{X} \) as
$$
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
$$
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Interpretations and optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
The function
$$
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
$$
can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
$$
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
$$
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
<p>
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
$$
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
which results in
$$
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
$$
or in a matrix-vector form as
$$
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right).
$$
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Interpretations and optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
We can rewrite
$$
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),
$$
as
$$
\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},
$$
and if the matrix \( \hat{X}^T\hat{X} \) is invertible we have the solution
$$
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
$$
The residuals \( \hat{\epsilon} \) are in turn given by
$$
\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
$$
and with
$$
\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
$$
we have
$$
\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
$$
meaning that the solution for \( \hat{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">The singular value decompostion </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Here we derive the equations for the SVD.
</div>
<p>
<!-- ------------------- end of main content --------------- -->
doc/pub/Regression/html/Regression.html
+149 -51
View File
@@ -75,7 +75,16 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec3'),
('Optimizing our parameters', 2, None, '___sec4')]}
('Optimizing our parameters', 2, None, '___sec4'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec5'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec6'),
('The singular value decompostion', 2, None, '___sec7')]}
end of tocinfo -->
<body>
@@ -90,8 +99,8 @@ MathJax.Hub.Config({
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
@@ -117,7 +126,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 14, 2017</h4></center> <!-- date -->
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -172,19 +181,13 @@ where \( \epsilon_i \) is the error in our approximation.
<p>
For every set of values \( y_i,x_i \) we have thus the corresponding set of equations
$$
\begin{align}
y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0
\label{_auto1}\\
y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1
\label{_auto2}\\
y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2
\label{_auto3}\\
\dots & \dots
\label{_auto4}\\
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.
\label{_auto5}\\
\label{_auto6}
\end{align}
\begin{align*}
y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\
y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\
y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
\dots & \dots \\
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
Defining the vectors
@@ -229,23 +232,15 @@ $$
We are obviously not limited to the above polynomial. We could replace the various powers of \( x \) with elements of Fourier series, that is, instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions.
For every set of values \( y_i,x_i \) we can then generalize the equations to
$$
\begin{align}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0
\label{_auto7}\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1
\label{_auto8}\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1
\label{_auto9}\\
\dots & \dots
\label{_auto10}\\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1
\label{_auto11}\\
\dots & \dots
\label{_auto12}\\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.
\label{_auto13}\\
\label{_auto14}
\end{align}
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
\dots & \dots \\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
We redefine in turn the matrix \( \hat{X} \) as
@@ -276,28 +271,131 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
We have defined the matrix \( \hat{X} \)
$$
\begin{align}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0
\label{_auto15}\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1
\label{_auto16}\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1
\label{_auto17}\\
\dots & \dots
\label{_auto18}\\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1
\label{_auto19}\\
\dots & \dots
\label{_auto20}\\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.
\label{_auto21}\\
\label{_auto22}
\end{align}
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
\dots & \dots \\
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
\end{align*}
$$
We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
$$
\hat{\tilde{y}}= \hat{X}\hat{\beta},
$$
and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parametrized values \( \tilde{y}_i \), namely
$$
Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
$$
or using the matrix \( \hat{X} \) as
$$
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
$$
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Interpretations and optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
The function
$$
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
$$
can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
$$
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
$$
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
<p>
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
$$
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
which results in
$$
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
$$
or in a matrix-vector form as
$$
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right).
$$
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Interpretations and optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
We can rewrite
$$
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),
$$
as
$$
\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},
$$
and if the matrix \( \hat{X}^T\hat{X} \) is invertible we have the solution
$$
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
$$
The residuals \( \hat{\epsilon} \) are in turn given by
$$
\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
$$
and with
$$
\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
$$
we have
$$
\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
$$
meaning that the solution for \( \hat{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">The singular value decompostion </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Here we derive the equations for the SVD.
</div>
<p>
<!-- ------------------- end of main content --------------- -->
doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/Regression/pdf/Regression-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/Regression/pdf/Regression-minted.pdf
BIN
View File
Binary file not shown.
doc/src/Regression/Regression.do.txt
+63 -10
View File
@@ -37,13 +37,13 @@ where $\epsilon_i$ is the error in our approximation.
!bblock
For every set of values $y_i,x_i$ we have thus the corresponding set of equations
!bt
\begin{align}
\begin{align*}
y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\
y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\
y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
\dots & \dots \\
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
\end{align}
\end{align*}
!et
Defining the vectors
!bt
@@ -89,7 +89,7 @@ we can rewrite our equations as
We are obviously not limited to the above polynomial. We could replace the various powers of $x$ with elements of Fourier series, that is, instead of $x_i^j$ we could have $\cos{(j x_i)}$ or $\sin{(j x_i)}$, or time series or other orthogonal functions.
For every set of values $y_i,x_i$ we can then generalize the equations to
!bt
\begin{align}
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
@@ -97,7 +97,7 @@ y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilo
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
\end{align}
\end{align*}
!et
We redefine in turn the matrix $\hat{X}$ as
!bt
@@ -127,7 +127,7 @@ The left-hand side of this equation forms know. Our error vector $\hat{\epsilon}
!bblock
We have defined the matrix $\hat{X}$
!bt
\begin{align}
\begin{align*}
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
@@ -135,7 +135,7 @@ y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilo
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
\dots & \dots \\
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
\end{align}
\end{align*}
!et
We well use this matrix to define the approximation $\hat{\tilde{y}}$ via the unknown quantity $\hat{\beta}$ as
!bt
@@ -146,13 +146,13 @@ We well use this matrix to define the approximation $\hat{\tilde{y}}$ via the un
and in order to find the optimal parameters $\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parametrized values $\tilde{y}_i$, namely
!bt
\[
Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left \hat{y}-\hat{\tilde{y}}\right)^T\left \hat{y}-\hat{\tilde{y}}\right),
Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
\]
!et
or using the matrix $\hat{X}$ as
!bt
\[
Q(\hat{\beta})=\left \hat{y}-\hat{X}\hat{\beta}\right)^T\left \hat{y}-\hat{X}\hat{\beta}\right).
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
\]
!et
!eblock
@@ -164,7 +164,7 @@ Q(\hat{\beta})=\left \hat{y}-\hat{X}\hat{\beta}\right)^T\left \hat{y}-\hat{X}\ha
The function
!bt
\[
Q(\hat{\beta})=\left \hat{y}-\hat{X}\hat{\beta}\right)^T\left \hat{y}-\hat{X}\hat{\beta}\right),
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
\]
!et
can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value
@@ -178,7 +178,7 @@ where $\langle y_i \rangle$ is the mean value. Keep in mind also that till now
In order to find the parameters $\beta_i$ we will then minimize the spread of $Q(\hat{\beta})$ by requiring
!bt
\[
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{ }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
\]
!et
which results in
@@ -196,3 +196,56 @@ or in a matrix-vector form as
!eblock
!split
===== Interpretations and optimizing our parameters =====
!bblock
We can rewrite
!bt
\[
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),
\]
!et
as
!bt
\[
\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},
\]
!et
and if the matrix $\hat{X}^T\hat{X}$ is invertible we have the solution
!bt
\[
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
\]
!et
The residuals $\hat{\epsilon}$ are in turn given by
!bt
\[
\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
\]
!et
and with
!bt
\[
\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
\]
!et
we have
!bt
\[
\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
\]
!et
meaning that the solution for $\hat{\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.
!eblock
!split
===== The singular value decompostion =====
!bblock
Here we derive the equations for the SVD.
!eblock