Update on regression analysis
This commit is contained in:
mhjensen
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -139,7 +156,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
|
||||
<center><h4>Oct 18, 2017</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -161,6 +178,9 @@ MathJax.Hub.Config({
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -157,6 +174,10 @@ A regression model aims at finding a likelihood function \( p(y\vert \hat{x}) \)
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs002.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -138,6 +155,8 @@ where \( \epsilon_i \) is the error in our approximation.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
@@ -151,6 +170,11 @@ where \( \epsilon_i \) is the error in our approximation.
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs003.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -134,36 +151,6 @@ y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
|
||||
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
Defining the vectors
|
||||
$$
|
||||
\hat{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T,
|
||||
$$
|
||||
|
||||
$$
|
||||
\hat{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T,
|
||||
$$
|
||||
|
||||
$$
|
||||
\hat{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T,
|
||||
$$
|
||||
|
||||
and the matrix
|
||||
$$
|
||||
\hat{X}=
|
||||
\begin{bmatrix}
|
||||
1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\
|
||||
1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\
|
||||
1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\
|
||||
\dots& \dots &\dots& \dots & \dots &\dots\\
|
||||
1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\
|
||||
\end{bmatrix}
|
||||
$$
|
||||
|
||||
we can rewrite our equations as
|
||||
$$
|
||||
\hat{y} = \hat{X}\hat{\beta}+\hat{\epsilon}.
|
||||
$$
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -182,6 +169,10 @@ $$
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs004.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -120,42 +137,39 @@ MathJax.Hub.Config({
|
||||
<a name="part0004"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec3" class="anchor">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<h2 id="___sec3" class="anchor">Rewriting the fitting procedure as a linear algebra problem, follows </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
We are obviously not limited to the above polynomial. We could replace the various powers of \( x \) with elements of Fourier series, that is, instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions.
|
||||
For every set of values \( y_i,x_i \) we can then generalize the equations to
|
||||
Defining the vectors
|
||||
$$
|
||||
\begin{align*}
|
||||
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
|
||||
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
|
||||
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
|
||||
\dots & \dots \\
|
||||
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
|
||||
\dots & \dots \\
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
\hat{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T,
|
||||
$$
|
||||
|
||||
We redefine in turn the matrix \( \hat{X} \) as
|
||||
$$
|
||||
\hat{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T,
|
||||
$$
|
||||
|
||||
$$
|
||||
\hat{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T,
|
||||
$$
|
||||
|
||||
and the matrix
|
||||
$$
|
||||
\hat{X}=
|
||||
\begin{bmatrix}
|
||||
x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\
|
||||
x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\
|
||||
x_{20}& x_{21}^1 &x_{22}2& \dots & \dots &x_{2,n-1}\\
|
||||
1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\
|
||||
1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\
|
||||
1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\
|
||||
\dots& \dots &\dots& \dots & \dots &\dots\\
|
||||
x_{n-1,00}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\
|
||||
1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\
|
||||
\end{bmatrix}
|
||||
$$
|
||||
|
||||
and without loss of generality we rewrite again our equations as
|
||||
we can rewrite our equations as
|
||||
$$
|
||||
\hat{y} = \hat{X}\hat{\beta}+\hat{\epsilon}.
|
||||
$$
|
||||
|
||||
The left-hand side of this equation forms know. Our error vector \( \hat{\epsilon} \) and the parameter vector \( \hat{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values?
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -174,6 +188,10 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs005.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -120,37 +137,23 @@ MathJax.Hub.Config({
|
||||
<a name="part0005"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec4" class="anchor">Optimizing our parameters </h2>
|
||||
<h2 id="___sec4" class="anchor">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
We have defined the matrix \( \hat{X} \)
|
||||
We are obviously not limited to the above polynomial. We could replace the various powers of \( x \) with elements of Fourier series, that is, instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions.
|
||||
For every set of values \( y_i,x_i \) we can then generalize the equations to
|
||||
$$
|
||||
\begin{align*}
|
||||
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
|
||||
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
|
||||
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
|
||||
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
|
||||
\dots & \dots \\
|
||||
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
|
||||
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
|
||||
\dots & \dots \\
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
|
||||
$$
|
||||
\hat{\tilde{y}}= \hat{X}\hat{\beta},
|
||||
$$
|
||||
|
||||
and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parametrized values \( \tilde{y}_i \), namely
|
||||
$$
|
||||
Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
|
||||
$$
|
||||
|
||||
or using the matrix \( \hat{X} \) as
|
||||
$$
|
||||
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
|
||||
$$
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -169,6 +172,10 @@ $$
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs006.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -120,39 +137,28 @@ MathJax.Hub.Config({
|
||||
<a name="part0006"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec5" class="anchor">Interpretations and optimizing our parameters </h2>
|
||||
<h2 id="___sec5" class="anchor">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
The function
|
||||
We redefine in turn the matrix \( \hat{X} \) as
|
||||
$$
|
||||
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
|
||||
\hat{X}=
|
||||
\begin{bmatrix}
|
||||
x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\
|
||||
x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\
|
||||
x_{20}& x_{21}^1 &x_{22}2& \dots & \dots &x_{2,n-1}\\
|
||||
\dots& \dots &\dots& \dots & \dots &\dots\\
|
||||
x_{n-1,00}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\
|
||||
\end{bmatrix}
|
||||
$$
|
||||
|
||||
can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
|
||||
and without loss of generality we rewrite again our equations as
|
||||
$$
|
||||
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
\hat{y} = \hat{X}\hat{\beta}+\hat{\epsilon}.
|
||||
$$
|
||||
|
||||
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
|
||||
|
||||
<p>
|
||||
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
|
||||
$$
|
||||
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
$$
|
||||
|
||||
which results in
|
||||
$$
|
||||
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
$$
|
||||
|
||||
or in a matrix-vector form as
|
||||
$$
|
||||
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right).
|
||||
$$
|
||||
|
||||
<p>
|
||||
The left-hand side of this equation forms know. Our error vector \( \hat{\epsilon} \) and the parameter vector \( \hat{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values?
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -171,6 +177,10 @@ $$
|
||||
<li class="active"><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs007.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -120,43 +137,22 @@ MathJax.Hub.Config({
|
||||
<a name="part0007"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec6" class="anchor">Interpretations and optimizing our parameters </h2>
|
||||
<h2 id="___sec6" class="anchor">Optimizing our parameters </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
We can rewrite
|
||||
We have defined the matrix \( \hat{X} \)
|
||||
$$
|
||||
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),
|
||||
\begin{align*}
|
||||
y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
|
||||
y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
|
||||
y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
|
||||
\dots & \dots \\
|
||||
y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
|
||||
\dots & \dots \\
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
$$
|
||||
|
||||
as
|
||||
$$
|
||||
\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},
|
||||
$$
|
||||
|
||||
and if the matrix \( \hat{X}^T\hat{X} \) is invertible we have the solution
|
||||
$$
|
||||
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
|
||||
$$
|
||||
|
||||
The residuals \( \hat{\epsilon} \) are in turn given by
|
||||
$$
|
||||
\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
|
||||
$$
|
||||
|
||||
and with
|
||||
$$
|
||||
\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
|
||||
$$
|
||||
|
||||
we have
|
||||
$$
|
||||
\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
|
||||
$$
|
||||
|
||||
meaning that the solution for \( \hat{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.
|
||||
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -175,6 +171,10 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li class="active"><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs008.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -120,17 +137,29 @@ MathJax.Hub.Config({
|
||||
<a name="part0008"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec7" class="anchor">The singular value decompostion </h2>
|
||||
<h2 id="___sec7" class="anchor">Optimizing our parameters, more details </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
Here we derive the equations for the SVD.
|
||||
We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
|
||||
$$
|
||||
\hat{\tilde{y}}= \hat{X}\hat{\beta},
|
||||
$$
|
||||
|
||||
and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parametrized values \( \tilde{y}_i \), namely
|
||||
$$
|
||||
Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
|
||||
$$
|
||||
|
||||
or using the matrix \( \hat{X} \) as
|
||||
$$
|
||||
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
|
||||
$$
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
@@ -144,6 +173,11 @@ Here we derive the equations for the SVD.
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li class="active"><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs009.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
|
||||
('General linear models', 2, None, '___sec1'),
|
||||
('Rewriting the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0009"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec8" class="anchor">Interpretations and optimizing our parameters </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
The function
|
||||
$$
|
||||
Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
|
||||
$$
|
||||
|
||||
can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
|
||||
$$
|
||||
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
$$
|
||||
|
||||
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
|
||||
|
||||
<p>
|
||||
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
|
||||
$$
|
||||
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
$$
|
||||
|
||||
which results in
|
||||
$$
|
||||
\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
$$
|
||||
|
||||
or in a matrix-vector form as
|
||||
$$
|
||||
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right).
|
||||
$$
|
||||
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Regression-bs008.html">«</a></li>
|
||||
<li><a href="._Regression-bs000.html">1</a></li>
|
||||
<li><a href="._Regression-bs001.html">2</a></li>
|
||||
<li><a href="._Regression-bs002.html">3</a></li>
|
||||
<li><a href="._Regression-bs003.html">4</a></li>
|
||||
<li><a href="._Regression-bs004.html">5</a></li>
|
||||
<li><a href="._Regression-bs005.html">6</a></li>
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li class="active"><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs010.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,206 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
|
||||
('General linear models', 2, None, '___sec1'),
|
||||
('Rewriting the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0010"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec9" class="anchor">Interpretations and optimizing our parameters </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
We can rewrite
|
||||
$$
|
||||
\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),
|
||||
$$
|
||||
|
||||
as
|
||||
$$
|
||||
\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},
|
||||
$$
|
||||
|
||||
and if the matrix \( \hat{X}^T\hat{X} \) is invertible we have the solution
|
||||
$$
|
||||
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Regression-bs009.html">«</a></li>
|
||||
<li><a href="._Regression-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Regression-bs002.html">3</a></li>
|
||||
<li><a href="._Regression-bs003.html">4</a></li>
|
||||
<li><a href="._Regression-bs004.html">5</a></li>
|
||||
<li><a href="._Regression-bs005.html">6</a></li>
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li class="active"><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs011.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
|
||||
('General linear models', 2, None, '___sec1'),
|
||||
('Rewriting the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0011"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec10" class="anchor">Interpretations and optimizing our parameters </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
The residuals \( \hat{\epsilon} \) are in turn given by
|
||||
$$
|
||||
\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
|
||||
$$
|
||||
|
||||
and with
|
||||
$$
|
||||
\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
|
||||
$$
|
||||
|
||||
we have
|
||||
$$
|
||||
\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
|
||||
$$
|
||||
|
||||
meaning that the solution for \( \hat{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.
|
||||
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Regression-bs010.html">«</a></li>
|
||||
<li><a href="._Regression-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Regression-bs003.html">4</a></li>
|
||||
<li><a href="._Regression-bs004.html">5</a></li>
|
||||
<li><a href="._Regression-bs005.html">6</a></li>
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li class="active"><a href="._Regression-bs011.html">12</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs012.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="http://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
|
||||
('General linear models', 2, None, '___sec1'),
|
||||
('Rewriting the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0012"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec11" class="anchor">The singular value decompostion </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then
|
||||
$$
|
||||
\hat{A} = \hat{U}\hat{\Sigma}\hat{V}
|
||||
$$
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Regression-bs011.html">«</a></li>
|
||||
<li><a href="._Regression-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Regression-bs004.html">5</a></li>
|
||||
<li><a href="._Regression-bs005.html">6</a></li>
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="._Regression-bs010.html">11</a></li>
|
||||
<li><a href="._Regression-bs011.html">12</a></li>
|
||||
<li class="active"><a href="._Regression-bs012.html">13</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -46,20 +46,33 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -74,8 +87,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -100,11 +113,15 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The singular value decompostion</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -139,7 +156,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
|
||||
<center><h4>Oct 18, 2017</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -161,6 +178,9 @@ MathJax.Hub.Config({
|
||||
<li><a href="._Regression-bs006.html">7</a></li>
|
||||
<li><a href="._Regression-bs007.html">8</a></li>
|
||||
<li><a href="._Regression-bs008.html">9</a></li>
|
||||
<li><a href="._Regression-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Regression-bs012.html">13</a></li>
|
||||
<li><a href="._Regression-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
\
|
||||
<!DOCTYPE html>
|
||||
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
@@ -116,8 +117,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -147,7 +148,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p> <br>
|
||||
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
|
||||
<center><h4>Oct 18, 2017</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -219,7 +220,15 @@ y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\e
|
||||
\end{align*}
|
||||
$$
|
||||
<p> <br>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec3">Rewriting the fitting procedure as a linear algebra problem, follows </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
Defining the vectors
|
||||
<p> <br>
|
||||
$$
|
||||
@@ -264,7 +273,7 @@ $$
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec3">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<h2 id="___sec4">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -283,7 +292,15 @@ y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}
|
||||
\end{align*}
|
||||
$$
|
||||
<p> <br>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec5">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
We redefine in turn the matrix \( \hat{X} \) as
|
||||
<p> <br>
|
||||
$$
|
||||
@@ -311,7 +328,7 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec4">Optimizing our parameters </h2>
|
||||
<h2 id="___sec6">Optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -329,7 +346,15 @@ y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,
|
||||
\end{align*}
|
||||
$$
|
||||
<p> <br>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec7">Optimizing our parameters, more details </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
|
||||
<p> <br>
|
||||
$$
|
||||
@@ -355,7 +380,7 @@ $$
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec5">Interpretations and optimizing our parameters </h2>
|
||||
<h2 id="___sec8">Interpretations and optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -403,7 +428,7 @@ $$
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec6">Interpretations and optimizing our parameters </h2>
|
||||
<h2 id="___sec9">Interpretations and optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -428,6 +453,16 @@ $$
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec10">Interpretations and optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
The residuals \( \hat{\epsilon} \) are in turn given by
|
||||
<p> <br>
|
||||
$$
|
||||
@@ -457,11 +492,16 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec7">The singular value decompostion </h2>
|
||||
<h2 id="___sec11">The singular value decompostion </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
Here we derive the equations for the SVD.
|
||||
How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then
|
||||
<p> <br>
|
||||
$$
|
||||
\hat{A} = \hat{U}\hat{\Sigma}\hat{V}
|
||||
$$
|
||||
<p> <br>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
@@ -66,20 +66,33 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -94,8 +107,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -121,7 +134,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
|
||||
<center><h4>Oct 18, 2017</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -168,6 +181,8 @@ where \( \epsilon_i \) is the error in our approximation.
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec2">Rewriting the fitting procedure as a linear algebra problem </h2>
|
||||
@@ -184,7 +199,16 @@ y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
|
||||
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
$$
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec3">Rewriting the fitting procedure as a linear algebra problem, follows </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
Defining the vectors
|
||||
$$
|
||||
\hat{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T,
|
||||
@@ -220,7 +244,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec3">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<h2 id="___sec4">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -237,7 +261,16 @@ y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsi
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
$$
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec5">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
We redefine in turn the matrix \( \hat{X} \) as
|
||||
$$
|
||||
\hat{X}=
|
||||
@@ -262,7 +295,7 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec4">Optimizing our parameters </h2>
|
||||
<h2 id="___sec6">Optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -278,7 +311,16 @@ y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsi
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
$$
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec7">Optimizing our parameters, more details </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
|
||||
$$
|
||||
\hat{\tilde{y}}= \hat{X}\hat{\beta},
|
||||
@@ -299,7 +341,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec5">Interpretations and optimizing our parameters </h2>
|
||||
<h2 id="___sec8">Interpretations and optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -338,7 +380,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec6">Interpretations and optimizing our parameters </h2>
|
||||
<h2 id="___sec9">Interpretations and optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -357,6 +399,17 @@ $$
|
||||
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
|
||||
$$
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec10">Interpretations and optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
The residuals \( \hat{\epsilon} \) are in turn given by
|
||||
$$
|
||||
\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
|
||||
@@ -381,11 +434,14 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec7">The singular value decompostion </h2>
|
||||
<h2 id="___sec11">The singular value decompostion </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
Here we derive the equations for the SVD.
|
||||
How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then
|
||||
$$
|
||||
\hat{A} = \hat{U}\hat{\Sigma}\hat{V}
|
||||
$$
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
@@ -71,20 +71,33 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
2,
|
||||
None,
|
||||
'___sec2'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
('Rewriting the fitting procedure as a linear algebra problem, follows',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Optimizing our parameters', 2, None, '___sec4'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('Generalizing the fitting procedure as a linear algebra problem',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Optimizing our parameters', 2, None, '___sec6'),
|
||||
('Optimizing our parameters, more details', 2, None, '___sec7'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('The singular value decompostion', 2, None, '___sec7')]}
|
||||
'___sec8'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Interpretations and optimizing our parameters',
|
||||
2,
|
||||
None,
|
||||
'___sec10'),
|
||||
('The singular value decompostion', 2, None, '___sec11')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -99,8 +112,8 @@ MathJax.Hub.Config({
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript"
|
||||
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
@@ -126,7 +139,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 17, 2017</h4></center> <!-- date -->
|
||||
<center><h4>Oct 18, 2017</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -173,6 +186,8 @@ where \( \epsilon_i \) is the error in our approximation.
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec2">Rewriting the fitting procedure as a linear algebra problem </h2>
|
||||
@@ -189,7 +204,16 @@ y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
|
||||
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
$$
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec3">Rewriting the fitting procedure as a linear algebra problem, follows </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
Defining the vectors
|
||||
$$
|
||||
\hat{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T,
|
||||
@@ -225,7 +249,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec3">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<h2 id="___sec4">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -242,7 +266,16 @@ y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsi
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
$$
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec5">Generalizing the fitting procedure as a linear algebra problem </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
We redefine in turn the matrix \( \hat{X} \) as
|
||||
$$
|
||||
\hat{X}=
|
||||
@@ -267,7 +300,7 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec4">Optimizing our parameters </h2>
|
||||
<h2 id="___sec6">Optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -283,7 +316,16 @@ y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsi
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
$$
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec7">Optimizing our parameters, more details </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
|
||||
$$
|
||||
\hat{\tilde{y}}= \hat{X}\hat{\beta},
|
||||
@@ -304,7 +346,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec5">Interpretations and optimizing our parameters </h2>
|
||||
<h2 id="___sec8">Interpretations and optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -343,7 +385,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec6">Interpretations and optimizing our parameters </h2>
|
||||
<h2 id="___sec9">Interpretations and optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -362,6 +404,17 @@ $$
|
||||
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
|
||||
$$
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec10">Interpretations and optimizing our parameters </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
The residuals \( \hat{\epsilon} \) are in turn given by
|
||||
$$
|
||||
\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
|
||||
@@ -386,11 +439,14 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec7">The singular value decompostion </h2>
|
||||
<h2 id="___sec11">The singular value decompostion </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
Here we derive the equations for the SVD.
|
||||
How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then
|
||||
$$
|
||||
\hat{A} = \hat{U}\hat{\Sigma}\hat{V}
|
||||
$$
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -32,6 +32,8 @@ y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_i x_i^j+
|
||||
where $\epsilon_i$ is the error in our approximation.
|
||||
|
||||
!eblock
|
||||
|
||||
|
||||
!split
|
||||
===== Rewriting the fitting procedure as a linear algebra problem =====
|
||||
!bblock
|
||||
@@ -45,6 +47,12 @@ y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
|
||||
y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
!et
|
||||
!eblock
|
||||
|
||||
|
||||
!split
|
||||
===== Rewriting the fitting procedure as a linear algebra problem, follows =====
|
||||
!bblock
|
||||
Defining the vectors
|
||||
!bt
|
||||
\[
|
||||
@@ -99,6 +107,12 @@ y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsi
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
!et
|
||||
!eblock
|
||||
|
||||
|
||||
!split
|
||||
===== Generalizing the fitting procedure as a linear algebra problem =====
|
||||
!bblock
|
||||
We redefine in turn the matrix $\hat{X}$ as
|
||||
!bt
|
||||
\[
|
||||
@@ -137,6 +151,12 @@ y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsi
|
||||
y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
|
||||
\end{align*}
|
||||
!et
|
||||
!eblock
|
||||
|
||||
|
||||
!split
|
||||
===== Optimizing our parameters, more details =====
|
||||
!bblock
|
||||
We well use this matrix to define the approximation $\hat{\tilde{y}}$ via the unknown quantity $\hat{\beta}$ as
|
||||
!bt
|
||||
\[
|
||||
@@ -219,6 +239,12 @@ and if the matrix $\hat{X}^T\hat{X}$ is invertible we have the solution
|
||||
\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
|
||||
\]
|
||||
!et
|
||||
|
||||
!eblock
|
||||
|
||||
!split
|
||||
===== Interpretations and optimizing our parameters =====
|
||||
!bblock
|
||||
The residuals $\hat{\epsilon}$ are in turn given by
|
||||
!bt
|
||||
\[
|
||||
@@ -242,10 +268,16 @@ meaning that the solution for $\hat{\beta}$ is the one which minimizes the resid
|
||||
!eblock
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== The singular value decompostion =====
|
||||
!bblock
|
||||
Here we derive the equations for the SVD.
|
||||
How can we use the singular value decomposition to find the parameters $\beta_j$? More details will come. We first note that a general $m\times n$ matrix $\hat{A}$ can be written in terms of a diagonal matrix $\hat{\Sigma}$ of dimensionality $n\times n$ and two orthognal matrices $\hat{U}$ and $\hat{V}$, where the first has dimensionality $m \times n$ and the last dimensionality $n\times n$. We have then
|
||||
!bt
|
||||
\[
|
||||
\hat{A} = \hat{U}\hat{\Sigma}\hat{V}
|
||||
\]
|
||||
!et
|
||||
!eblock
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user