lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on regression analysis

Browse Source
This commit is contained in:
mhjensen
2018-09-06 11:28:45 +02:00
parent eb9bc8a6ea
commit f5b9eb32a2
56 changed files with 9374 additions and 5675 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Regression/html/._Regression-bs037.html
+143 -64
View File
@@ -41,17 +41,14 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('Regression analysis, overarching aims', 2, None, '___sec0'),
('General linear models', 2, None, '___sec1'),
('Regression analysis, overarching aims II', 2, None, '___sec1'),
('General linear models', 2, None, '___sec2'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec2'),
'___sec3'),
('Rewriting the fitting procedure as a linear algebra problem, '
'follows',
2,
None,
'___sec3'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec4'),
@@ -59,12 +56,12 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec5'),
('Optimizing our parameters', 2, None, '___sec6'),
('Optimizing our parameters, more details', 2, None, '___sec7'),
('Interpretations and optimizing our parameters',
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec8'),
'___sec6'),
('Optimizing our parameters', 2, None, '___sec7'),
('Optimizing our parameters, more details', 2, None, '___sec8'),
('Interpretations and optimizing our parameters',
2,
None,
@@ -73,47 +70,71 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec10'),
('The $\\chi^2$ function', 2, None, '___sec11'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec11'),
('The $\\chi^2$ function', 2, None, '___sec12'),
('The $\\chi^2$ function', 2, None, '___sec13'),
('The $\\chi^2$ function', 2, None, '___sec14'),
('The $\\chi^2$ function', 2, None, '___sec15'),
('The $\\chi^2$ function', 2, None, '___sec16'),
('Simple regression model', 2, None, '___sec17'),
('The $\\chi^2$ function', 2, None, '___sec17'),
('Simple regression model', 2, None, '___sec18'),
('Simple regression model, now using _scikit-learn_',
2,
None,
'___sec18'),
'___sec19'),
('Simple linear regression model using _scikit-learn_',
2,
None,
'___sec19'),
('Simple linear regression model', 2, None, '___sec20'),
('Less noise', 2, None, '___sec21'),
('How to study our fits', 2, None, '___sec22'),
('Minimizing the cost function', 2, None, '___sec23'),
('Relative error', 2, None, '___sec24'),
('The richness of _scikit-learn_', 2, None, '___sec25'),
('Functions in _scikit-learn_', 2, None, '___sec26'),
('Other functions in _scikit-learn_', 2, None, '___sec27'),
'___sec20'),
('Simple linear regression model', 2, None, '___sec21'),
('Less noise', 2, None, '___sec22'),
('How to study our fits', 2, None, '___sec23'),
('Minimizing the cost function', 2, None, '___sec24'),
('Relative error', 2, None, '___sec25'),
('The richness of _scikit-learn_', 2, None, '___sec26'),
('Functions in _scikit-learn_', 2, None, '___sec27'),
('Other functions in _scikit-learn_', 2, None, '___sec28'),
('The mean absolute error and other functions in _scikit-learn_',
2,
None,
'___sec28'),
('Cubic polynomial in _scikit-learn_', 2, None, '___sec29'),
('Simple regression model with gradient descent',
'___sec29'),
('Cubic polynomial in _scikit-learn_', 2, None, '___sec30'),
('Polynomial Regression', 2, None, '___sec31'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
None,
'___sec30'),
('Simple regression model with stochastic gradient descent',
2,
None,
'___sec31'),
('Polynomial Regression', 2, None, '___sec32'),
('Ridge and Lasso Regression', 2, None, '___sec33'),
'___sec32'),
('Expectation value and variance', 2, None, '___sec33'),
('The singular value decompostion', 2, None, '___sec34'),
('Lasso and Ridge regression', 2, None, '___sec35'),
('Logistic regression', 2, None, '___sec36')]}
('Code examples for Ridge and Lasso Regression',
2,
None,
'___sec35'),
('From standard regression to Ridge regressions',
2,
None,
'___sec36'),
('Fixing the singularity', 2, None, '___sec37'),
('A second-order polynomial with Ridge and Lasso',
2,
None,
'___sec38'),
('Fitting vs. predicting when data is in the model class',
2,
None,
'___sec39'),
('Fitting versus predicting when data is not in the model class',
2,
None,
'___sec40'),
('The code', 2, None, '___sec41'),
('Generating test data', 2, None, '___sec42'),
('Lasso regression', 2, None, '___sec43'),
('Logistic regression', 2, None, '___sec44')]}
end of tocinfo -->
<body>
@@ -152,42 +173,50 @@ MathJax.Hub.Config({
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">Regression analysis, overarching aims II</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, follows</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs013.html#___sec12" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs014.html#___sec13" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs015.html#___sec14" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs016.html#___sec15" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs017.html#___sec16" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs018.html#___sec17" style="font-size: 80%;">Simple regression model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs019.html#___sec18" style="font-size: 80%;">Simple regression model, now using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs020.html#___sec19" style="font-size: 80%;">Simple linear regression model using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs021.html#___sec20" style="font-size: 80%;">Simple linear regression model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs022.html#___sec21" style="font-size: 80%;">Less noise</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs023.html#___sec22" style="font-size: 80%;">How to study our fits</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs024.html#___sec23" style="font-size: 80%;">Minimizing the cost function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs025.html#___sec24" style="font-size: 80%;">Relative error</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs026.html#___sec25" style="font-size: 80%;">The richness of <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs027.html#___sec26" style="font-size: 80%;">Functions in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs028.html#___sec27" style="font-size: 80%;">Other functions in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs029.html#___sec28" style="font-size: 80%;">The mean absolute error and other functions in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs030.html#___sec29" style="font-size: 80%;">Cubic polynomial in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs031.html#___sec30" style="font-size: 80%;">Simple regression model with gradient descent</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs032.html#___sec31" style="font-size: 80%;">Simple regression model with stochastic gradient descent</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs033.html#___sec32" style="font-size: 80%;">Polynomial Regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs034.html#___sec33" style="font-size: 80%;">Ridge and Lasso Regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs018.html#___sec17" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs019.html#___sec18" style="font-size: 80%;">Simple regression model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs020.html#___sec19" style="font-size: 80%;">Simple regression model, now using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs021.html#___sec20" style="font-size: 80%;">Simple linear regression model using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs022.html#___sec21" style="font-size: 80%;">Simple linear regression model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs023.html#___sec22" style="font-size: 80%;">Less noise</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs024.html#___sec23" style="font-size: 80%;">How to study our fits</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs025.html#___sec24" style="font-size: 80%;">Minimizing the cost function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs026.html#___sec25" style="font-size: 80%;">Relative error</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs027.html#___sec26" style="font-size: 80%;">The richness of <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs028.html#___sec27" style="font-size: 80%;">Functions in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs029.html#___sec28" style="font-size: 80%;">Other functions in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs030.html#___sec29" style="font-size: 80%;">The mean absolute error and other functions in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs031.html#___sec30" style="font-size: 80%;">Cubic polynomial in <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs032.html#___sec31" style="font-size: 80%;">Polynomial Regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs033.html#___sec32" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs034.html#___sec33" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs035.html#___sec34" style="font-size: 80%;">The singular value decompostion</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs036.html#___sec35" style="font-size: 80%;">Lasso and Ridge regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec36" style="font-size: 80%;">Logistic regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs036.html#___sec35" style="font-size: 80%;">Code examples for Ridge and Lasso Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec36" style="font-size: 80%;">From standard regression to Ridge regressions</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs038.html#___sec37" style="font-size: 80%;">Fixing the singularity</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs039.html#___sec38" style="font-size: 80%;">A second-order polynomial with Ridge and Lasso</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs040.html#___sec39" style="font-size: 80%;">Fitting vs. predicting when data is in the model class</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs041.html#___sec40" style="font-size: 80%;">Fitting versus predicting when data is not in the model class</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs042.html#___sec41" style="font-size: 80%;">The code</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs043.html#___sec42" style="font-size: 80%;">Generating test data</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs044.html#___sec43" style="font-size: 80%;">Lasso regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs045.html#___sec44" style="font-size: 80%;">Logistic regression</a></li>
</ul>
</li>
@@ -203,14 +232,55 @@ MathJax.Hub.Config({
<a name="part0037"></a>
<!-- !split -->
<h2 id="___sec36" class="anchor">Logistic regression </h2>
Add discussion about classification versus regression, show examples of more than two cases and why regression is not the best approach. Motivate for k-nearest neighbors
<h2 id="___sec36" class="anchor">From standard regression to Ridge regressions </h2>
<p>
Add examples on classification problems
One of the typical problems we encounter with linear regression, in particular
when the matrix \( \hat{X} \) (our so-called design matrix) is high-dimensional,
are problems with near singular or singular matrices. The column vectors of \( \hat{X} \)
may be linearly dependent, normally referred to as super-collinearity.
This means that the matrix may be rank deficient and it is basically impossible to
to model the data using linear regression. As an example, consider the matrix
$$
\begin{align*}
\mathbf{X} & = \left[
\begin{array}{rrr}
1 & -1 & 2
\\
1 & 0 & 1
\\
1 & 2 & -1
\\
1 & 1 & 0
\end{array} \right]
\end{align*}
$$
<p>
The columns of \( \hat{X} \) are linearly dependent. We se this easily since the
the first column is the row-wise sum of the other two columns. The rank (more correct,
the column rank) of a matrix is the dimension of the space spanned by the
column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number
of linearly independent columns. In this particular case the matrix has rank 2.
<p>
Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies
that the inverse of the matrix \( \hat{X}^T\hat{x} \) (the matrix we needto invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this
$$
\begin{align*}
\hat{X} & = \left[
\begin{array}{rr}
1 & -1
\\
1 & -1
\end{array} \right].
\end{align*}
$$
We see easily that \( \mbox{det}(\hat{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined.
This is equivalent to saying that the matrix \( \hat{X} \) has at least an eigenvalue which is zero.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
@@ -226,6 +296,15 @@ Add examples on classification problems
<li><a href="._Regression-bs035.html">36</a></li>
<li><a href="._Regression-bs036.html">37</a></li>
<li class="active"><a href="._Regression-bs037.html">38</a></li>
<li><a href="._Regression-bs038.html">39</a></li>
<li><a href="._Regression-bs039.html">40</a></li>
<li><a href="._Regression-bs040.html">41</a></li>
<li><a href="._Regression-bs041.html">42</a></li>
<li><a href="._Regression-bs042.html">43</a></li>
<li><a href="._Regression-bs043.html">44</a></li>
<li><a href="._Regression-bs044.html">45</a></li>
<li><a href="._Regression-bs045.html">46</a></li>
<li><a href="._Regression-bs038.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->