lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e71b95a3d7b05c656277ddf69bd8fed0aedb0178
FYS-STK4155/doc/pub/Regression/html/._Regression-bs038.html
T
mhjensen 9d1de4186e updated regression analysis slides
2018-09-07 05:32:45 +02:00

339 lines
19 KiB
HTML
Raw Blame History

<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
('Regression analysis, overarching aims II', 2, None, '___sec1'),
('General linear models', 2, None, '___sec2'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec3'),
('Rewriting the fitting procedure as a linear algebra problem, '
'follows',
2,
None,
'___sec4'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec5'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec6'),
('Optimizing our parameters', 2, None, '___sec7'),
('Optimizing our parameters, more details', 2, None, '___sec8'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec9'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec10'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec11'),
('The $\\chi^2$ function', 2, None, '___sec12'),
('The $\\chi^2$ function', 2, None, '___sec13'),
('The $\\chi^2$ function', 2, None, '___sec14'),
('The $\\chi^2$ function', 2, None, '___sec15'),
('The $\\chi^2$ function', 2, None, '___sec16'),
('The $\\chi^2$ function', 2, None, '___sec17'),
('Simple regression model', 2, None, '___sec18'),
('Simple regression model, now using _scikit-learn_',
2,
None,
'___sec19'),
('Simple linear regression model using _scikit-learn_',
2,
None,
'___sec20'),
('Simple linear regression model', 2, None, '___sec21'),
('Less noise', 2, None, '___sec22'),
('How to study our fits', 2, None, '___sec23'),
('Minimizing the cost function', 2, None, '___sec24'),
('Relative error', 2, None, '___sec25'),
('The richness of _scikit-learn_', 2, None, '___sec26'),
('Functions in _scikit-learn_', 2, None, '___sec27'),
('Other functions in _scikit-learn_', 2, None, '___sec28'),
('The mean absolute error and other functions in _scikit-learn_',
2,
None,
'___sec29'),
('Cubic polynomial in _scikit-learn_', 2, None, '___sec30'),
('Polynomial Regression', 2, None, '___sec31'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
None,
'___sec32'),
('Expectation value and variance', 2, None, '___sec33'),
('The singular value decompostion', 2, None, '___sec34'),
('From standard regression to Ridge regressions',
2,
None,
'___sec35'),
('Fixing the singularity', 2, None, '___sec36'),
('Fitting vs. predicting when data is in the model class',
2,
None,
'___sec37'),
('Fitting versus predicting when data is not in the model class',
2,
None,
'___sec38'),
('An example code without the model assessment part',
2,
None,
'___sec39'),
('Generating test data', 2, None, '___sec40'),
('How can we effectively evaluate the various models?',
2,
None,
'___sec41'),
('Code examples for Ridge and Lasso Regression',
2,
None,
'___sec42'),
('A second-order polynomial with Ridge and Lasso',
2,
None,
'___sec43'),
('Resampling methods', 2, None, '___sec44'),
('Resampling approaches can be computationally expensive',
2,
None,
'___sec45'),
('Log-likelihood', 2, None, '___sec46'),
('Cross-validation', 2, None, '___sec47'),
('Computationally expensive', 2, None, '___sec48'),
('Various steps in cross-validation', 2, None, '___sec49'),
('How to set up the cross-validation for Ridge and/or Lasso',
2,
None,
'___sec50'),
('Predicted Residual Error Sum of Squares', 2, None, '___sec51'),
('Bootstrap', 2, None, '___sec52')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">Regression analysis, overarching aims II</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs013.html#___sec12" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs014.html#___sec13" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs015.html#___sec14" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs016.html#___sec15" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs017.html#___sec16" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs018.html#___sec17" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs019.html#___sec18" style="font-size: 80%;">Simple regression model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs020.html#___sec19" style="font-size: 80%;">Simple regression model, now using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs021.html#___sec20" style="font-size: 80%;">Simple linear regression model using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs022.html#___sec21" style="font-size: 80%;">Simple linear regression model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs023.html#___sec22" style="font-size: 80%;">Less noise</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs024.html#___sec23" style="font-size: 80%;">How to study our fits</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs025.html#___sec24" style="font-size: 80%;">Minimizing the cost function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs026.html#___sec25" style="font-size: 80%;">Relative error</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs027.html#___sec26" style="font-size: 80%;">The richness of <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs028.html#___sec27" style="font-size: 80%;">Functions in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs029.html#___sec28" style="font-size: 80%;">Other functions in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs030.html#___sec29" style="font-size: 80%;">The mean absolute error and other functions in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs031.html#___sec30" style="font-size: 80%;">Cubic polynomial in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs032.html#___sec31" style="font-size: 80%;">Polynomial Regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs033.html#___sec32" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs034.html#___sec33" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs035.html#___sec34" style="font-size: 80%;">The singular value decompostion</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs036.html#___sec35" style="font-size: 80%;">From standard regression to Ridge regressions</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs037.html#___sec36" style="font-size: 80%;">Fixing the singularity</a></li>
<!-- navigation toc: --> <li><a href="#___sec37" style="font-size: 80%;">Fitting vs. predicting when data is in the model class</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs039.html#___sec38" style="font-size: 80%;">Fitting versus predicting when data is not in the model class</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs040.html#___sec39" style="font-size: 80%;">An example code without the model assessment part</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs041.html#___sec40" style="font-size: 80%;">Generating test data</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs042.html#___sec41" style="font-size: 80%;">How can we effectively evaluate the various models?</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs043.html#___sec42" style="font-size: 80%;">Code examples for Ridge and Lasso Regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs044.html#___sec43" style="font-size: 80%;">A second-order polynomial with Ridge and Lasso</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs045.html#___sec44" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs046.html#___sec45" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs047.html#___sec46" style="font-size: 80%;">Log-likelihood</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs048.html#___sec47" style="font-size: 80%;">Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs049.html#___sec48" style="font-size: 80%;">Computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs050.html#___sec49" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs051.html#___sec50" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs052.html#___sec51" style="font-size: 80%;">Predicted Residual Error Sum of Squares</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs053.html#___sec52" style="font-size: 80%;">Bootstrap</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0038"></a>
<!-- !split -->
<h2 id="___sec37" class="anchor">Fitting vs. predicting when data is in the model class </h2>
<p>
We start by considering the case
\( f(x)=2x \).
<p>
Then the data is clearly generated by a model that is contained within
all three model classes we are using to make predictions (linear
models, third order polynomials, and tenth order polynomials).
<p>
Run the code for the following cases:
<ol>
<li> For \( f(x)=2x \) , \( Ntrain=10 \) and \( \sigma =0 \) (noiseless case), train the three classes of models (linear, third-order polynomial, and tenth order polynomial) for a training set when \( x \in [0,1] \) . Make graphs comparing fits for different order of polynomials. Which model fits the data the best?</li>
<li> Do you think that the data that has the least error on the training set will also make the best predictions? Why or why not? Can you try to discuss and formalize your intuition? What can go right and what can go wrong?</li>
<li> Check your answer by seeing how well your fits predict newly generated test data (including on data outside the range you fit on, for example \( x \in [0,1.2] \) ) using the code below. How well do you do on points in the range of x where you trained the model? How about points outside the original training data set?</li>
<li> Repeat the above for \( f(x)=2x \) , \( Ntrain=10 \) , and \( \sigma=1 \) . What changes?</li>
</ol>
Repeat the exercises above for \( f(x)=2x \) , \( Ntrain=100 \) , and \( \sigma=1 \) . What changes?
Summarize what you have learned about the relationship between model complexity (number of parameters), goodness of fit on training data, and the ability to predict well.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._Regression-bs037.html">&laquo;</a></li>
<li><a href="._Regression-bs000.html">1</a></li>
<li><a href="">...</a></li>
<li><a href="._Regression-bs030.html">31</a></li>
<li><a href="._Regression-bs031.html">32</a></li>
<li><a href="._Regression-bs032.html">33</a></li>
<li><a href="._Regression-bs033.html">34</a></li>
<li><a href="._Regression-bs034.html">35</a></li>
<li><a href="._Regression-bs035.html">36</a></li>
<li><a href="._Regression-bs036.html">37</a></li>
<li><a href="._Regression-bs037.html">38</a></li>
<li class="active"><a href="._Regression-bs038.html">39</a></li>
<li><a href="._Regression-bs039.html">40</a></li>
<li><a href="._Regression-bs040.html">41</a></li>
<li><a href="._Regression-bs041.html">42</a></li>
<li><a href="._Regression-bs042.html">43</a></li>
<li><a href="._Regression-bs043.html">44</a></li>
<li><a href="._Regression-bs044.html">45</a></li>
<li><a href="._Regression-bs045.html">46</a></li>
<li><a href="._Regression-bs046.html">47</a></li>
<li><a href="._Regression-bs047.html">48</a></li>
<li><a href="">...</a></li>
<li><a href="._Regression-bs053.html">54</a></li>
<li><a href="._Regression-bs039.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink