lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e3e369b98fcdf88d5c603ea36734dd2fd79b67c2
FYS-STK4155/doc/pub/Regression/html/._Regression-bs099.html
T
mhjensen 3fdabd3871 revising regression
2019-06-10 22:10:42 -04:00

651 lines
44 KiB
HTML
Raw Blame History

<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis">
<title>Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Why Linear Regression (aka Ordinary Least Squares and family)',
2,
None,
'___sec0'),
('Regression analysis, overarching aims', 2, None, '___sec1'),
('Regression analysis, overarching aims II', 2, None, '___sec2'),
('Examples', 2, None, '___sec3'),
('General linear models', 2, None, '___sec4'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec5'),
('Rewriting the fitting procedure as a linear algebra problem, '
'more details',
2,
None,
'___sec6'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec7'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec8'),
('Optimizing our parameters', 2, None, '___sec9'),
('Our model for the nuclear binding energies',
2,
None,
'___sec10'),
('Optimizing our parameters, more details', 2, None, '___sec11'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec12'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec13'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec14'),
('Own code for Ordinary Least Squares', 2, None, '___sec15'),
('Adding error analysis and training set up',
2,
None,
'___sec16'),
('The $\\chi^2$ function', 2, None, '___sec17'),
('The $\\chi^2$ function', 2, None, '___sec18'),
('The $\\chi^2$ function', 2, None, '___sec19'),
('The $\\chi^2$ function', 2, None, '___sec20'),
('The $\\chi^2$ function', 2, None, '___sec21'),
('The $\\chi^2$ function', 2, None, '___sec22'),
('Fitting an Equation of State for Dense Nuclear Matter',
2,
None,
'___sec23'),
('The code', 2, None, '___sec24'),
('Splitting our Data in Training and Test data',
2,
None,
'___sec25'),
('The singular value decomposition', 2, None, '___sec26'),
('The Ising model', 2, None, '___sec27'),
('Reformulating the problem to suit regression',
2,
None,
'___sec28'),
('Linear regression', 2, None, '___sec29'),
('Singular Value decomposition', 2, None, '___sec30'),
('Linear Regression Problems', 2, None, '___sec31'),
('Fixing the singularity', 2, None, '___sec32'),
('Basic math of the SVD', 2, None, '___sec33'),
('The SVD, a Fantastic Algorithm', 2, None, '___sec34'),
('Another Example', 2, None, '___sec35'),
('Economy-size SVD', 2, None, '___sec36'),
('Mathematical Properties', 2, None, '___sec37'),
('Ridge and LASSO Regression', 2, None, '___sec38'),
('More on Ridge Regression', 2, None, '___sec39'),
('Interpreting the Ridge results', 2, None, '___sec40'),
('More interpretations', 2, None, '___sec41'),
('Where are we going?', 2, None, '___sec42'),
('Resampling methods', 2, None, '___sec43'),
('Resampling approaches can be computationally expensive',
2,
None,
'___sec44'),
('Why resampling methods ?', 2, None, '___sec45'),
('Statistical analysis', 2, None, '___sec46'),
('Statistics', 2, None, '___sec47'),
('Statistics, moments', 2, None, '___sec48'),
('Statistics, central moments', 2, None, '___sec49'),
('Statistics, covariance', 2, None, '___sec50'),
('Statistics, more covariance', 2, None, '___sec51'),
('Statistics, independent variables', 2, None, '___sec52'),
('Statistics, more variance', 2, None, '___sec53'),
('Statistics and stochastic processes', 2, None, '___sec54'),
('Statistics and sample variables', 2, None, '___sec55'),
('Statistics, sample variance and covariance',
2,
None,
'___sec56'),
('Statistics, law of large numbers', 2, None, '___sec57'),
('Statistics, more on sample error', 2, None, '___sec58'),
('Statistics', 2, None, '___sec59'),
('Statistics, central limit theorem', 2, None, '___sec60'),
('Statistics, more technicalities', 2, None, '___sec61'),
('Statistics', 2, None, '___sec62'),
('Statistics and sample variance', 2, None, '___sec63'),
('Statistics, uncorrelated results', 2, None, '___sec64'),
('Statistics, computations', 2, None, '___sec65'),
('Statistics, more on computations of errors',
2,
None,
'___sec66'),
('Statistics, wrapping up 1', 2, None, '___sec67'),
('Statistics, final expression', 2, None, '___sec68'),
('Statistics, effective number of correlations',
2,
None,
'___sec69'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
None,
'___sec70'),
('Assumptions made', 2, None, '___sec71'),
('Expectation value and variance', 2, None, '___sec72'),
('Expectation value and variance for $\\boldsymbol{\\beta}$',
2,
None,
'___sec73'),
('Cross-validation', 2, None, '___sec74'),
('Computationally expensive', 2, None, '___sec75'),
('Various steps in cross-validation', 2, None, '___sec76'),
('How to set up the cross-validation for Ridge and/or Lasso',
2,
None,
'___sec77'),
('Resampling methods: Jackknife and Bootstrap',
2,
None,
'___sec78'),
('Resampling methods: Jackknife', 2, None, '___sec79'),
('Jackknife code example', 2, None, '___sec80'),
('Resampling methods: Bootstrap', 2, None, '___sec81'),
('Resampling methods: Bootstrap background', 2, None, '___sec82'),
('Resampling methods: More Bootstrap background',
2,
None,
'___sec83'),
('Resampling methods: Bootstrap approach', 2, None, '___sec84'),
('Resampling methods: Bootstrap steps', 2, None, '___sec85'),
('Code example for the Bootstrap method', 2, None, '___sec86'),
('Code Example for Cross-validation and $k$-fold '
'Cross-validation',
2,
None,
'___sec87'),
('The bias-variance tradeoff', 2, None, '___sec88'),
('Example code for Bias-Variance tradeoff', 2, None, '___sec89'),
('Understanding what happens', 2, None, '___sec90'),
('Summing up', 2, None, '___sec91'),
("Another Example rom Scikit-Learn's Repository",
2,
None,
'___sec92'),
('The one-dimensional Ising model', 2, None, '___sec93'),
('Ridge regression', 2, None, '___sec94'),
('LASSO regression', 2, None, '___sec95'),
('Performance as function of the regularization parameter',
2,
None,
'___sec96'),
('Finding the optimal value of $\\lambda$', 2, None, '___sec97'),
('Further Exercises', 2, None, '___sec98'),
('Exercise 1', 3, None, '___sec99'),
('Exercise 2, variance of the parameters $\\beta$ in linear '
'regression',
3,
None,
'___sec100'),
('Exercise 3', 3, None, '___sec101'),
('Exercise 4', 3, None, '___sec102')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="Regression-bs.html">Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;"><b>Why Linear Regression (aka Ordinary Least Squares and family)</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;"><b>Regression analysis, overarching aims</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;"><b>Regression analysis, overarching aims II</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;"><b>Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;"><b>General linear models</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;"><b>Rewriting the fitting procedure as a linear algebra problem</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;"><b>Rewriting the fitting procedure as a linear algebra problem, more details</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;"><b>Generalizing the fitting procedure as a linear algebra problem</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;"><b>Generalizing the fitting procedure as a linear algebra problem</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;"><b>Optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;"><b>Our model for the nuclear binding energies</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;"><b>Optimizing our parameters, more details</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs013.html#___sec12" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs014.html#___sec13" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs015.html#___sec14" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs016.html#___sec15" style="font-size: 80%;"><b>Own code for Ordinary Least Squares</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs017.html#___sec16" style="font-size: 80%;"><b>Adding error analysis and training set up</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs018.html#___sec17" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs019.html#___sec18" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs020.html#___sec19" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs021.html#___sec20" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs022.html#___sec21" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs023.html#___sec22" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs024.html#___sec23" style="font-size: 80%;"><b>Fitting an Equation of State for Dense Nuclear Matter</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs025.html#___sec24" style="font-size: 80%;"><b>The code</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs026.html#___sec25" style="font-size: 80%;"><b>Splitting our Data in Training and Test data</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs027.html#___sec26" style="font-size: 80%;"><b>The singular value decomposition</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs028.html#___sec27" style="font-size: 80%;"><b>The Ising model</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs029.html#___sec28" style="font-size: 80%;"><b>Reformulating the problem to suit regression</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs030.html#___sec29" style="font-size: 80%;"><b>Linear regression</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs031.html#___sec30" style="font-size: 80%;"><b>Singular Value decomposition</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs032.html#___sec31" style="font-size: 80%;"><b>Linear Regression Problems</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs033.html#___sec32" style="font-size: 80%;"><b>Fixing the singularity</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs034.html#___sec33" style="font-size: 80%;"><b>Basic math of the SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs035.html#___sec34" style="font-size: 80%;"><b>The SVD, a Fantastic Algorithm</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs036.html#___sec35" style="font-size: 80%;"><b>Another Example</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs037.html#___sec36" style="font-size: 80%;"><b>Economy-size SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs038.html#___sec37" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs039.html#___sec38" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs040.html#___sec39" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs041.html#___sec40" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs042.html#___sec41" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs043.html#___sec42" style="font-size: 80%;"><b>Where are we going?</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs044.html#___sec43" style="font-size: 80%;"><b>Resampling methods</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs045.html#___sec44" style="font-size: 80%;"><b>Resampling approaches can be computationally expensive</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs046.html#___sec45" style="font-size: 80%;"><b>Why resampling methods ?</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs047.html#___sec46" style="font-size: 80%;"><b>Statistical analysis</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs048.html#___sec47" style="font-size: 80%;"><b>Statistics</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs049.html#___sec48" style="font-size: 80%;"><b>Statistics, moments</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs050.html#___sec49" style="font-size: 80%;"><b>Statistics, central moments</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs051.html#___sec50" style="font-size: 80%;"><b>Statistics, covariance</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs052.html#___sec51" style="font-size: 80%;"><b>Statistics, more covariance</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs053.html#___sec52" style="font-size: 80%;"><b>Statistics, independent variables</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs054.html#___sec53" style="font-size: 80%;"><b>Statistics, more variance</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs055.html#___sec54" style="font-size: 80%;"><b>Statistics and stochastic processes</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs056.html#___sec55" style="font-size: 80%;"><b>Statistics and sample variables</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs057.html#___sec56" style="font-size: 80%;"><b>Statistics, sample variance and covariance</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs058.html#___sec57" style="font-size: 80%;"><b>Statistics, law of large numbers</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs059.html#___sec58" style="font-size: 80%;"><b>Statistics, more on sample error</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs060.html#___sec59" style="font-size: 80%;"><b>Statistics</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs061.html#___sec60" style="font-size: 80%;"><b>Statistics, central limit theorem</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs062.html#___sec61" style="font-size: 80%;"><b>Statistics, more technicalities</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs063.html#___sec62" style="font-size: 80%;"><b>Statistics</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs064.html#___sec63" style="font-size: 80%;"><b>Statistics and sample variance</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs065.html#___sec64" style="font-size: 80%;"><b>Statistics, uncorrelated results</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs066.html#___sec65" style="font-size: 80%;"><b>Statistics, computations</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs067.html#___sec66" style="font-size: 80%;"><b>Statistics, more on computations of errors</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs068.html#___sec67" style="font-size: 80%;"><b>Statistics, wrapping up 1</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs069.html#___sec68" style="font-size: 80%;"><b>Statistics, final expression</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs070.html#___sec69" style="font-size: 80%;"><b>Statistics, effective number of correlations</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs071.html#___sec70" style="font-size: 80%;"><b>Linking the regression analysis with a statistical interpretation</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs072.html#___sec71" style="font-size: 80%;"><b>Assumptions made</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs073.html#___sec72" style="font-size: 80%;"><b>Expectation value and variance</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs074.html#___sec73" style="font-size: 80%;"><b>Expectation value and variance for \( \boldsymbol{\beta} \)</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs075.html#___sec74" style="font-size: 80%;"><b>Cross-validation</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs076.html#___sec75" style="font-size: 80%;"><b>Computationally expensive</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs077.html#___sec76" style="font-size: 80%;"><b>Various steps in cross-validation</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs078.html#___sec77" style="font-size: 80%;"><b>How to set up the cross-validation for Ridge and/or Lasso</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs079.html#___sec78" style="font-size: 80%;"><b>Resampling methods: Jackknife and Bootstrap</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs080.html#___sec79" style="font-size: 80%;"><b>Resampling methods: Jackknife</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs081.html#___sec80" style="font-size: 80%;"><b>Jackknife code example</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs082.html#___sec81" style="font-size: 80%;"><b>Resampling methods: Bootstrap</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs083.html#___sec82" style="font-size: 80%;"><b>Resampling methods: Bootstrap background</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs084.html#___sec83" style="font-size: 80%;"><b>Resampling methods: More Bootstrap background</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs085.html#___sec84" style="font-size: 80%;"><b>Resampling methods: Bootstrap approach</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs086.html#___sec85" style="font-size: 80%;"><b>Resampling methods: Bootstrap steps</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs087.html#___sec86" style="font-size: 80%;"><b>Code example for the Bootstrap method</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs088.html#___sec87" style="font-size: 80%;"><b>Code Example for Cross-validation and \( k \)-fold Cross-validation</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs089.html#___sec88" style="font-size: 80%;"><b>The bias-variance tradeoff</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs090.html#___sec89" style="font-size: 80%;"><b>Example code for Bias-Variance tradeoff</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs091.html#___sec90" style="font-size: 80%;"><b>Understanding what happens</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs092.html#___sec91" style="font-size: 80%;"><b>Summing up</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs093.html#___sec92" style="font-size: 80%;"><b>Another Example rom Scikit-Learn's Repository</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs094.html#___sec93" style="font-size: 80%;"><b>The one-dimensional Ising model</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs095.html#___sec94" style="font-size: 80%;"><b>Ridge regression</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs096.html#___sec95" style="font-size: 80%;"><b>LASSO regression</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs097.html#___sec96" style="font-size: 80%;"><b>Performance as function of the regularization parameter</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs098.html#___sec97" style="font-size: 80%;"><b>Finding the optimal value of \( \lambda \)</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec98" style="font-size: 80%;"><b>Further Exercises</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec99" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise 1</a></li>
<!-- navigation toc: --> <li><a href="#___sec100" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise 2, variance of the parameters \( \beta \) in linear regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec101" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise 3</a></li>
<!-- navigation toc: --> <li><a href="#___sec102" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise 4</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0099"></a>
<!-- !split -->
<h2 id="___sec98" class="anchor">Further Exercises </h2>
<h3 id="___sec99" class="anchor">Exercise 1 </h3>
<p>
We will generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal {N}(0,1) \).
The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">5*</span>x<span style="color: #666666">*</span>x<span style="color: #666666">+0.1*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
</pre></div>
<ol>
<li> Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial.</li>
<li> Use thereafter <b>scikit-learn</b> (see again the examples in the regression slides) and compare with your own code.</li>
<li> Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</li>
</ol>
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
$$
and the \( R^2 \) score function.
If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
$$
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
$$
where we have defined the mean value of \( \hat{y} \) as
$$
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
$$
<p>
You can use the functionality included in scikit-learn. If you feel
for it, you can use your own program and define functions which
compute the above two functions. Discuss the meaning of these
results. Try also to vary the coefficient in front of the added
stochastic noise term and discuss the quality of the fits.
<h3 id="___sec100" class="anchor">Exercise 2, variance of the parameters \( \beta \) in linear regression </h3>
<p>
Show that the variance of the parameters \( \beta \) in the linear regression method (chapter 3, equation (3.8) of <a href="https://www.springer.com/gp/book/9780387848570" target="_self">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a>) is given as
$$
\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2,
$$
with
$$
\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2,
$$
where we have assumed that we fit a function of degree \( p-1 \) (for example a polynomial in \( x \)).
<h3 id="___sec101" class="anchor">Exercise 3 </h3>
<p>
This exercise is a continuation of exercise 1. We will
use the same function to generate our data set, still staying with a
simple function \( y(x) \) which we want to fit using linear regression,
but now extending the analysis to include the Ridge and the Lasso
regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods.
<p>
We will thus again generate our own dataset for a function \( y(x) \) where
\( x \in [0,1] \) and defined by random numbers computed with the uniform
distribution. The function \( y \) is a quadratic polynomial in \( x \) with
added stochastic noise according to the normal distribution \( \cal{N}(0,1) \).
<p>
The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">5*</span>x<span style="color: #666666">*</span>x<span style="color: #666666">+0.1*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
</pre></div>
<ol>
<li> Write your own code for the Ridge method and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 1. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).</li>
<li> Repeat the above but using the functionality of <b>scikit-learn</b>. Compare your code with the results from <b>scikit-learn</b>. Remember to run with the same random numbers for generating \( x \) and \( y \).</li>
<li> Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of <b>scikit-learn</b> and compute their variances. Discuss the results of these variances as functions</li>
<li> Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results.</li>
<li> Try to implement the cross-validation as well.</li>
<li> Finally, using <b>scikit-learn</b> or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</li>
</ol>
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
$$
and the \( R^2 \) score function.
If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
$$
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
$$
where we have defined the mean value of \( \hat{y} \) as
$$
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
$$
Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods.
<h3 id="___sec102" class="anchor">Exercise 4 </h3>
<p>
We will study how
to fit polynomials to a specific two-dimensional function called
<a href="http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf" target="_self">Franke's
function</a>. This
is a function which has been widely used when testing various interpolation and fitting
algorithms. Furthermore, after having established the model and the
method, we will employ resamling techniques such as the cross-validation and/or
the bootstrap methods, in order to perform a proper assessment of our models.
<p>
The Franke function, which is a weighted sum of four exponentials reads as follows
$$
\begin{align*}
f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\
&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }.
\end{align*}
$$
<p>
The function will be defined for \( x,y\in [0,1] \). Our first step will
be to perform an OLS regression analysis of this function, trying out
a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y,
x^2, y^2, xy, \dots] \). We will also include cross-validation and
bootstrap as resampling techniques. As in homeworks 1 and 2, we
can use a uniform distribution to set up the arrays of values for \( x \)
and \( y \), or as in the example below just a fix values for \( x \) and \( y \) with a given step size.
In this case we will have two predictors and need to fit a
function (for example a polynomial) of \( x \) and \( y \). Thereafter we will
repeat much of the same procedure using the the Ridge and
Lasso regression methods, introducing thus a dependence on the bias
(penalty) \( \lambda \).
<p>
The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> Axes3D
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">matplotlib</span> <span style="color: #008000; font-weight: bold">import</span> cm
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">matplotlib.ticker</span> <span style="color: #008000; font-weight: bold">import</span> LinearLocator, FormatStrFormatter
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">random</span> <span style="color: #008000; font-weight: bold">import</span> random, seed
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>gca(projection<span style="color: #666666">=</span><span style="color: #BA2121">&#39;3d&#39;</span>)
<span style="color: #408080; font-style: italic"># Make data.</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">0</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0.05</span>)
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">0</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0.05</span>)
x, y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>meshgrid(x,y)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">FrankeFunction</span>(x,y):
term1 <span style="color: #666666">=</span> <span style="color: #666666">0.75*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">0.25*</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>) <span style="color: #666666">-</span> <span style="color: #666666">0.25*</span>((<span style="color: #666666">9*</span>y<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>))
term2 <span style="color: #666666">=</span> <span style="color: #666666">0.75*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>((<span style="color: #666666">9*</span>x<span style="color: #666666">+1</span>)<span style="color: #666666">**2</span>)<span style="color: #666666">/49.0</span> <span style="color: #666666">-</span> <span style="color: #666666">0.1*</span>(<span style="color: #666666">9*</span>y<span style="color: #666666">+1</span>))
term3 <span style="color: #666666">=</span> <span style="color: #666666">0.5*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-7</span>)<span style="color: #666666">**2/4.0</span> <span style="color: #666666">-</span> <span style="color: #666666">0.25*</span>((<span style="color: #666666">9*</span>y<span style="color: #666666">-3</span>)<span style="color: #666666">**2</span>))
term4 <span style="color: #666666">=</span> <span style="color: #666666">-0.2*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-4</span>)<span style="color: #666666">**2</span> <span style="color: #666666">-</span> (<span style="color: #666666">9*</span>y<span style="color: #666666">-7</span>)<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">return</span> term1 <span style="color: #666666">+</span> term2 <span style="color: #666666">+</span> term3 <span style="color: #666666">+</span> term4
z <span style="color: #666666">=</span> FrankeFunction(x, y)
<span style="color: #408080; font-style: italic"># Plot the surface.</span>
surf <span style="color: #666666">=</span> ax<span style="color: #666666">.</span>plot_surface(x, y, z, cmap<span style="color: #666666">=</span>cm<span style="color: #666666">.</span>coolwarm,
linewidth<span style="color: #666666">=0</span>, antialiased<span style="color: #666666">=</span><span style="color: #008000">False</span>)
<span style="color: #408080; font-style: italic"># Customize the z axis.</span>
ax<span style="color: #666666">.</span>set_zlim(<span style="color: #666666">-0.10</span>, <span style="color: #666666">1.40</span>)
ax<span style="color: #666666">.</span>zaxis<span style="color: #666666">.</span>set_major_locator(LinearLocator(<span style="color: #666666">10</span>))
ax<span style="color: #666666">.</span>zaxis<span style="color: #666666">.</span>set_major_formatter(FormatStrFormatter(<span style="color: #BA2121">&#39;</span><span style="color: #BB6688; font-weight: bold">%.02f</span><span style="color: #BA2121">&#39;</span>))
<span style="color: #408080; font-style: italic"># Add a color bar which maps values to colors.</span>
fig<span style="color: #666666">.</span>colorbar(surf, shrink<span style="color: #666666">=0.5</span>, aspect<span style="color: #666666">=5</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
We will thus again generate our own dataset for a function \( \mathrm{FrankeFunction}(x,y) \) where
\( x,y \in [0,1] \) could be defined by random numbers computed with the uniform
distribution. The function \( f(x,y) \) is the Franke function. You should explore also the addition
an added stochastic noise to this function using the normal distribution \( \cal{N}(0,1) \).
<p>
Write your own code (using either a matrix inversion or a singular value decomposition from e.g., <b>numpy</b> ) or use your code from exercises 1 and 3
and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the confidence intervals of the parameters \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE)
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
$$
and the \( R^2 \) score function.
If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
$$
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
$$
where we have defined the mean value of \( \hat{y} \) as
$$
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
$$
<p>
Perform a resampling of the data where you split the data in training data and test data. Implement the \( k \)-fold cross-validation algorithm and/or the bootstrap algorithm
and evaluate again the MSE and the \( R^2 \) functions resulting from the test data. Evaluate also the bias and variance of the final models.
<p>
Write then your own code for the Ridge method, either using matrix
inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the
previous exercise (for the same polynomials and include resampling
techniques) but now for different values of \( \lambda \). Compare and
analyze your results with those obtained with standard OLS. Study the
dependence on \( \lambda \) while also varying eventually the strength of
the noise in your expression for \( \mathrm{FrankeFunction}(x,y) \).
<p>
Then perform the same studies but now with Lasso regression. Use the functionalities of
<b>scikit-learn</b>. Give a critical discussion of the three methods and a
judgement of which model fits the data best.
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._Regression-bs098.html">&laquo;</a></li>
<li><a href="._Regression-bs000.html">1</a></li>
<li><a href="">...</a></li>
<li><a href="._Regression-bs091.html">92</a></li>
<li><a href="._Regression-bs092.html">93</a></li>
<li><a href="._Regression-bs093.html">94</a></li>
<li><a href="._Regression-bs094.html">95</a></li>
<li><a href="._Regression-bs095.html">96</a></li>
<li><a href="._Regression-bs096.html">97</a></li>
<li><a href="._Regression-bs097.html">98</a></li>
<li><a href="._Regression-bs098.html">99</a></li>
<li class="active"><a href="._Regression-bs099.html">100</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink