final update for lecture Thursday
This commit is contained in:
Morten Hjorth-Jensen
@@ -60,6 +60,10 @@ Automatically generated HTML file from DocOnce source
|
||||
2,
|
||||
None,
|
||||
'code-for-svd-and-inversion-of-matrices'),
|
||||
('Inverse of Rectangular Matrix',
|
||||
2,
|
||||
None,
|
||||
'inverse-of-rectangular-matrix'),
|
||||
('Ridge and LASSO Regression',
|
||||
2,
|
||||
None,
|
||||
@@ -274,65 +278,66 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs005.html#what-does-it-mean" style="font-size: 80%;">What does it mean?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs006.html#and-finally-boldsymbol-x-boldsymbol-x-t" style="font-size: 80%;">And finally \( \boldsymbol{X}\boldsymbol{X}^T \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs007.html#code-for-svd-and-inversion-of-matrices" style="font-size: 80%;">Code for SVD and Inversion of Matrices</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs008.html#ridge-and-lasso-regression" style="font-size: 80%;">Ridge and LASSO Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs009.html#from-ols-to-ridge-and-lasso" style="font-size: 80%;">From OLS to Ridge and Lasso</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs010.html#deriving-the-ridge-regression-equations" style="font-size: 80%;">Deriving the Ridge Regression Equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs011.html#svd-analysis" style="font-size: 80%;">SVD analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs012.html#interpreting-the-ridge-results" style="font-size: 80%;">Interpreting the Ridge results</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs013.html#more-interpretations" style="font-size: 80%;">More interpretations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs014.html#deriving-the-lasso-regression-equations" style="font-size: 80%;">Deriving the Lasso Regression Equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs015.html#simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression" style="font-size: 80%;">Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs016.html#ridge-regression" style="font-size: 80%;">Ridge Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs017.html#lasso-regression" style="font-size: 80%;">Lasso Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs018.html#yet-another-example" style="font-size: 80%;">Yet another Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs019.html#the-ols-case" style="font-size: 80%;">The OLS case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs020.html#the-ridge-case" style="font-size: 80%;">The Ridge case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs021.html#writing-the-cost-function" style="font-size: 80%;">Writing the Cost Function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs022.html#lasso-case" style="font-size: 80%;">Lasso case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs023.html#the-first-case" style="font-size: 80%;">The first Case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs024.html#simple-code-for-solving-the-above-problem" style="font-size: 80%;">Simple code for solving the above problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs025.html#with-lasso-regression" style="font-size: 80%;">With Lasso Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs026.html#another-example-now-with-a-polynomial-fit" style="font-size: 80%;">Another Example, now with a polynomial fit</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs027.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs028.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs029.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs030.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs031.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs032.html#independent-and-identically-distrubuted-iid" style="font-size: 80%;">Independent and Identically Distrubuted (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs033.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs034.html#a-new-cost-function" style="font-size: 80%;">A new Cost Function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs035.html#more-basic-statistics-and-bayes-theorem" style="font-size: 80%;">More basic Statistics and Bayes' theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs036.html#marginal-probability" style="font-size: 80%;">Marginal Probability</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs037.html#conditional-probability" style="font-size: 80%;">Conditional Probability</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs038.html#bayes-theorem" style="font-size: 80%;">Bayes' Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs039.html#frliday-september-10" style="font-size: 80%;">Frliday September 10</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs043.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs045.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs042.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs043.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs044.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs045.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs046.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs047.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs048.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs049.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs050.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs051.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs052.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs053.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs054.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs055.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs056.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs057.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs058.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs059.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs060.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs061.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs062.html#summing-up" style="font-size: 80%;">Summing up</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs063.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs064.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs065.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs066.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs008.html#inverse-of-rectangular-matrix" style="font-size: 80%;">Inverse of Rectangular Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs009.html#ridge-and-lasso-regression" style="font-size: 80%;">Ridge and LASSO Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs010.html#from-ols-to-ridge-and-lasso" style="font-size: 80%;">From OLS to Ridge and Lasso</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs011.html#deriving-the-ridge-regression-equations" style="font-size: 80%;">Deriving the Ridge Regression Equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs012.html#svd-analysis" style="font-size: 80%;">SVD analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs013.html#interpreting-the-ridge-results" style="font-size: 80%;">Interpreting the Ridge results</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs014.html#more-interpretations" style="font-size: 80%;">More interpretations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs015.html#deriving-the-lasso-regression-equations" style="font-size: 80%;">Deriving the Lasso Regression Equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs016.html#simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression" style="font-size: 80%;">Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs017.html#ridge-regression" style="font-size: 80%;">Ridge Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs018.html#lasso-regression" style="font-size: 80%;">Lasso Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs019.html#yet-another-example" style="font-size: 80%;">Yet another Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs020.html#the-ols-case" style="font-size: 80%;">The OLS case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs021.html#the-ridge-case" style="font-size: 80%;">The Ridge case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs022.html#writing-the-cost-function" style="font-size: 80%;">Writing the Cost Function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs023.html#lasso-case" style="font-size: 80%;">Lasso case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs024.html#the-first-case" style="font-size: 80%;">The first Case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs025.html#simple-code-for-solving-the-above-problem" style="font-size: 80%;">Simple code for solving the above problem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs026.html#with-lasso-regression" style="font-size: 80%;">With Lasso Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs027.html#another-example-now-with-a-polynomial-fit" style="font-size: 80%;">Another Example, now with a polynomial fit</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs028.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs029.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs030.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs031.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs032.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs033.html#independent-and-identically-distrubuted-iid" style="font-size: 80%;">Independent and Identically Distrubuted (iid)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs034.html#maximum-likelihood-estimation-mle" style="font-size: 80%;">Maximum Likelihood Estimation (MLE)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs035.html#a-new-cost-function" style="font-size: 80%;">A new Cost Function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs036.html#more-basic-statistics-and-bayes-theorem" style="font-size: 80%;">More basic Statistics and Bayes' theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs037.html#marginal-probability" style="font-size: 80%;">Marginal Probability</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs038.html#conditional-probability" style="font-size: 80%;">Conditional Probability</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs039.html#bayes-theorem" style="font-size: 80%;">Bayes' Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs040.html#frliday-september-10" style="font-size: 80%;">Frliday September 10</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs044.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs046.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs043.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs044.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs045.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs046.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs047.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs048.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs049.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs050.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs051.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs052.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs053.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs054.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs055.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs056.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs057.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs058.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs059.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs060.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs061.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs062.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs063.html#summing-up" style="font-size: 80%;">Summing up</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs064.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs065.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs066.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week36-bs067.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -391,7 +396,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._week36-bs008.html">9</a></li>
|
||||
<li><a href="._week36-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week36-bs066.html">67</a></li>
|
||||
<li><a href="._week36-bs067.html">68</a></li>
|
||||
<li><a href="._week36-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -201,7 +201,7 @@ $$
|
||||
Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
|
||||
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\end{bmatrix},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -366,17 +366,74 @@ Let us first look at a matrix which does not causes problems and write our own f
|
||||
|
||||
|
||||
<span style="color: #228B22">#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])</span>
|
||||
X = np.array( [ [<span style="color: #B452CD">1</span>,<span style="color: #B452CD">2</span>],[<span style="color: #B452CD">2</span>,<span style="color: #B452CD">3</span>]])
|
||||
<span style="color: #228B22"># Non-singular square matrix</span>
|
||||
X = np.array( [ [<span style="color: #B452CD">1</span>,<span style="color: #B452CD">2</span>,<span style="color: #B452CD">3</span>],[<span style="color: #B452CD">2</span>,<span style="color: #B452CD">4</span>,<span style="color: #B452CD">5</span>],[<span style="color: #B452CD">3</span>,<span style="color: #B452CD">5</span>,<span style="color: #B452CD">6</span>]])
|
||||
<span style="color: #658b00">print</span>(X)
|
||||
A = np.transpose(X) @ X
|
||||
<span style="color: #228B22"># Brute force inversion</span>
|
||||
B = np.linalg.inv(A)
|
||||
B = np.linalg.inv(A) <span style="color: #228B22"># here we could use np.linalg.pinv(A)</span>
|
||||
C = SVDinv(A)
|
||||
<span style="color: #658b00">print</span>(np.abs(B-C))
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="inverse-of-rectangular-matrix">Inverse of Rectangular Matrix </h2>
|
||||
|
||||
<p>
|
||||
Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular.
|
||||
|
||||
<p>
|
||||
The pseudoinverse is the generalization of the matrix inverse for square matrices to
|
||||
rectangular matrices where the number of rows and columns are not equal.
|
||||
|
||||
<p>
|
||||
It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.
|
||||
It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.
|
||||
|
||||
<p>
|
||||
Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \)
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #228B22"># SVD inversion</span>
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">SVDinv</span>(A):
|
||||
U, s, VT = np.linalg.svd(A)
|
||||
<span style="color: #228B22"># reciprocals of singular values of s</span>
|
||||
d = <span style="color: #B452CD">1.0</span> / s
|
||||
<span style="color: #228B22"># create m x n D matrix</span>
|
||||
D = np.zeros(A.shape)
|
||||
<span style="color: #228B22"># populate D with n x n diagonal matrix</span>
|
||||
D[:A.shape[<span style="color: #B452CD">1</span>], :A.shape[<span style="color: #B452CD">1</span>]] = np.diag(d)
|
||||
UT = np.transpose(U)
|
||||
V = np.transpose(VT)
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.matmul(V,np.matmul(D.T,UT))
|
||||
|
||||
|
||||
A = np.array([ [<span style="color: #B452CD">0.3</span>, <span style="color: #B452CD">0.4</span>], [<span style="color: #B452CD">0.5</span>, <span style="color: #B452CD">0.6</span>], [<span style="color: #B452CD">0.7</span>, <span style="color: #B452CD">0.8</span>],[<span style="color: #B452CD">0.9</span>, <span style="color: #B452CD">1.0</span>]])
|
||||
<span style="color: #658b00">print</span>(A)
|
||||
<span style="color: #228B22"># Brute force inversion of super-collinear matrix</span>
|
||||
B = np.linalg.pinv(A)
|
||||
<span style="color: #658b00">print</span>(B)
|
||||
<span style="color: #228B22"># Compare our own algorithm with pinv</span>
|
||||
C = SVDinv(A)
|
||||
<span style="color: #658b00">print</span>(np.abs(C-B))
|
||||
</pre></div>
|
||||
<p>
|
||||
As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by <b>Numpy</b>.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
|
||||
|
||||
|
||||
@@ -80,6 +80,10 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
2,
|
||||
None,
|
||||
'code-for-svd-and-inversion-of-matrices'),
|
||||
('Inverse of Rectangular Matrix',
|
||||
2,
|
||||
None,
|
||||
'inverse-of-rectangular-matrix'),
|
||||
('Ridge and LASSO Regression',
|
||||
2,
|
||||
None,
|
||||
@@ -333,7 +337,7 @@ $$
|
||||
|
||||
Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix
|
||||
$$
|
||||
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
|
||||
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\end{bmatrix},
|
||||
$$
|
||||
|
||||
where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is
|
||||
@@ -479,17 +483,72 @@ Let us first look at a matrix which does not causes problems and write our own f
|
||||
|
||||
|
||||
<span style="color: #228B22">#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])</span>
|
||||
X = np.array( [ [<span style="color: #B452CD">1</span>,<span style="color: #B452CD">2</span>],[<span style="color: #B452CD">2</span>,<span style="color: #B452CD">3</span>]])
|
||||
<span style="color: #228B22"># Non-singular square matrix</span>
|
||||
X = np.array( [ [<span style="color: #B452CD">1</span>,<span style="color: #B452CD">2</span>,<span style="color: #B452CD">3</span>],[<span style="color: #B452CD">2</span>,<span style="color: #B452CD">4</span>,<span style="color: #B452CD">5</span>],[<span style="color: #B452CD">3</span>,<span style="color: #B452CD">5</span>,<span style="color: #B452CD">6</span>]])
|
||||
<span style="color: #658b00">print</span>(X)
|
||||
A = np.transpose(X) @ X
|
||||
<span style="color: #228B22"># Brute force inversion</span>
|
||||
B = np.linalg.inv(A)
|
||||
B = np.linalg.inv(A) <span style="color: #228B22"># here we could use np.linalg.pinv(A)</span>
|
||||
C = SVDinv(A)
|
||||
<span style="color: #658b00">print</span>(np.abs(B-C))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="inverse-of-rectangular-matrix">Inverse of Rectangular Matrix </h2>
|
||||
|
||||
<p>
|
||||
Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular.
|
||||
|
||||
<p>
|
||||
The pseudoinverse is the generalization of the matrix inverse for square matrices to
|
||||
rectangular matrices where the number of rows and columns are not equal.
|
||||
|
||||
<p>
|
||||
It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.
|
||||
It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.
|
||||
|
||||
<p>
|
||||
Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \)
|
||||
$$
|
||||
\boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T,
|
||||
$$
|
||||
|
||||
where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #228B22"># SVD inversion</span>
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">SVDinv</span>(A):
|
||||
U, s, VT = np.linalg.svd(A)
|
||||
<span style="color: #228B22"># reciprocals of singular values of s</span>
|
||||
d = <span style="color: #B452CD">1.0</span> / s
|
||||
<span style="color: #228B22"># create m x n D matrix</span>
|
||||
D = np.zeros(A.shape)
|
||||
<span style="color: #228B22"># populate D with n x n diagonal matrix</span>
|
||||
D[:A.shape[<span style="color: #B452CD">1</span>], :A.shape[<span style="color: #B452CD">1</span>]] = np.diag(d)
|
||||
UT = np.transpose(U)
|
||||
V = np.transpose(VT)
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.matmul(V,np.matmul(D.T,UT))
|
||||
|
||||
|
||||
A = np.array([ [<span style="color: #B452CD">0.3</span>, <span style="color: #B452CD">0.4</span>], [<span style="color: #B452CD">0.5</span>, <span style="color: #B452CD">0.6</span>], [<span style="color: #B452CD">0.7</span>, <span style="color: #B452CD">0.8</span>],[<span style="color: #B452CD">0.9</span>, <span style="color: #B452CD">1.0</span>]])
|
||||
<span style="color: #658b00">print</span>(A)
|
||||
<span style="color: #228B22"># Brute force inversion of super-collinear matrix</span>
|
||||
B = np.linalg.pinv(A)
|
||||
<span style="color: #658b00">print</span>(B)
|
||||
<span style="color: #228B22"># Compare our own algorithm with pinv</span>
|
||||
C = SVDinv(A)
|
||||
<span style="color: #658b00">print</span>(np.abs(C-B))
|
||||
</pre></div>
|
||||
<p>
|
||||
As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by <b>Numpy</b>.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
|
||||
|
||||
<p>
|
||||
|
||||
@@ -85,6 +85,10 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
2,
|
||||
None,
|
||||
'code-for-svd-and-inversion-of-matrices'),
|
||||
('Inverse of Rectangular Matrix',
|
||||
2,
|
||||
None,
|
||||
'inverse-of-rectangular-matrix'),
|
||||
('Ridge and LASSO Regression',
|
||||
2,
|
||||
None,
|
||||
@@ -338,7 +342,7 @@ $$
|
||||
|
||||
Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix
|
||||
$$
|
||||
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
|
||||
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\end{bmatrix},
|
||||
$$
|
||||
|
||||
where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is
|
||||
@@ -484,17 +488,72 @@ Let us first look at a matrix which does not causes problems and write our own f
|
||||
|
||||
|
||||
<span style="color: #408080; font-style: italic">#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])</span>
|
||||
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ [<span style="color: #666666">1</span>,<span style="color: #666666">2</span>],[<span style="color: #666666">2</span>,<span style="color: #666666">3</span>]])
|
||||
<span style="color: #408080; font-style: italic"># Non-singular square matrix</span>
|
||||
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ [<span style="color: #666666">1</span>,<span style="color: #666666">2</span>,<span style="color: #666666">3</span>],[<span style="color: #666666">2</span>,<span style="color: #666666">4</span>,<span style="color: #666666">5</span>],[<span style="color: #666666">3</span>,<span style="color: #666666">5</span>,<span style="color: #666666">6</span>]])
|
||||
<span style="color: #008000">print</span>(X)
|
||||
A <span style="color: #666666">=</span> np<span style="color: #666666">.</span>transpose(X) <span style="color: #666666">@</span> X
|
||||
<span style="color: #408080; font-style: italic"># Brute force inversion</span>
|
||||
B <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(A)
|
||||
B <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(A) <span style="color: #408080; font-style: italic"># here we could use np.linalg.pinv(A)</span>
|
||||
C <span style="color: #666666">=</span> SVDinv(A)
|
||||
<span style="color: #008000">print</span>(np<span style="color: #666666">.</span>abs(B<span style="color: #666666">-</span>C))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="inverse-of-rectangular-matrix">Inverse of Rectangular Matrix </h2>
|
||||
|
||||
<p>
|
||||
Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular.
|
||||
|
||||
<p>
|
||||
The pseudoinverse is the generalization of the matrix inverse for square matrices to
|
||||
rectangular matrices where the number of rows and columns are not equal.
|
||||
|
||||
<p>
|
||||
It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.
|
||||
It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.
|
||||
|
||||
<p>
|
||||
Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \)
|
||||
$$
|
||||
\boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T,
|
||||
$$
|
||||
|
||||
where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #408080; font-style: italic"># SVD inversion</span>
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">SVDinv</span>(A):
|
||||
U, s, VT <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>svd(A)
|
||||
<span style="color: #408080; font-style: italic"># reciprocals of singular values of s</span>
|
||||
d <span style="color: #666666">=</span> <span style="color: #666666">1.0</span> <span style="color: #666666">/</span> s
|
||||
<span style="color: #408080; font-style: italic"># create m x n D matrix</span>
|
||||
D <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(A<span style="color: #666666">.</span>shape)
|
||||
<span style="color: #408080; font-style: italic"># populate D with n x n diagonal matrix</span>
|
||||
D[:A<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>], :A<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>]] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>diag(d)
|
||||
UT <span style="color: #666666">=</span> np<span style="color: #666666">.</span>transpose(U)
|
||||
V <span style="color: #666666">=</span> np<span style="color: #666666">.</span>transpose(VT)
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>matmul(V,np<span style="color: #666666">.</span>matmul(D<span style="color: #666666">.</span>T,UT))
|
||||
|
||||
|
||||
A <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([ [<span style="color: #666666">0.3</span>, <span style="color: #666666">0.4</span>], [<span style="color: #666666">0.5</span>, <span style="color: #666666">0.6</span>], [<span style="color: #666666">0.7</span>, <span style="color: #666666">0.8</span>],[<span style="color: #666666">0.9</span>, <span style="color: #666666">1.0</span>]])
|
||||
<span style="color: #008000">print</span>(A)
|
||||
<span style="color: #408080; font-style: italic"># Brute force inversion of super-collinear matrix</span>
|
||||
B <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(A)
|
||||
<span style="color: #008000">print</span>(B)
|
||||
<span style="color: #408080; font-style: italic"># Compare our own algorithm with pinv</span>
|
||||
C <span style="color: #666666">=</span> SVDinv(A)
|
||||
<span style="color: #008000">print</span>(np<span style="color: #666666">.</span>abs(C<span style="color: #666666">-</span>B))
|
||||
</pre></div>
|
||||
<p>
|
||||
As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by <b>Numpy</b>.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
|
||||
|
||||
<p>
|
||||
|
||||
Binary file not shown.
@@ -61,7 +61,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix},\n",
|
||||
"\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\end{bmatrix},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -310,11 +310,12 @@
|
||||
"\n",
|
||||
"\n",
|
||||
"#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])\n",
|
||||
"X = np.array( [ [1,2],[2,3]])\n",
|
||||
"# Non-singular square matrix\n",
|
||||
"X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])\n",
|
||||
"print(X)\n",
|
||||
"A = np.transpose(X) @ X\n",
|
||||
"# Brute force inversion\n",
|
||||
"B = np.linalg.inv(A)\n",
|
||||
"B = np.linalg.inv(A) # here we could use np.linalg.pinv(A)\n",
|
||||
"C = SVDinv(A)\n",
|
||||
"print(np.abs(B-C))"
|
||||
]
|
||||
@@ -323,6 +324,77 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Inverse of Rectangular Matrix\n",
|
||||
"\n",
|
||||
"Although our matrix to invert $\\boldsymbol{X}^T\\boldsymbol{X}$ is a square matrix, our matrix may be singular. \n",
|
||||
"\n",
|
||||
"The pseudoinverse is the generalization of the matrix inverse for square matrices to\n",
|
||||
"rectangular matrices where the number of rows and columns are not equal.\n",
|
||||
"\n",
|
||||
"It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.\n",
|
||||
"It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.\n",
|
||||
"\n",
|
||||
"Using the SVD we can obtain the pseudoinverse of a matrix $\\boldsymbol{A}$ (labeled here as $\\boldsymbol{A}_{\\mathrm{PI}}$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{A}_{\\mathrm{PI}}= \\boldsymbol{V}\\boldsymbol{D}_{\\mathrm{PI}}\\boldsymbol{U}^T,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"where $\\boldsymbol{D}_{\\mathrm{PI}}$ can be calculated by creating a diagonal matrix from $\\boldsymbol{Sigma}$ where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"# SVD inversion\n",
|
||||
"def SVDinv(A):\n",
|
||||
" U, s, VT = np.linalg.svd(A)\n",
|
||||
" # reciprocals of singular values of s\n",
|
||||
" d = 1.0 / s\n",
|
||||
" # create m x n D matrix\n",
|
||||
" D = np.zeros(A.shape)\n",
|
||||
" # populate D with n x n diagonal matrix\n",
|
||||
" D[:A.shape[1], :A.shape[1]] = np.diag(d)\n",
|
||||
" UT = np.transpose(U)\n",
|
||||
" V = np.transpose(VT)\n",
|
||||
" return np.matmul(V,np.matmul(D.T,UT))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"A = np.array([ [0.3, 0.4], [0.5, 0.6], [0.7, 0.8],[0.9, 1.0]])\n",
|
||||
"print(A)\n",
|
||||
"# Brute force inversion of super-collinear matrix\n",
|
||||
"B = np.linalg.pinv(A)\n",
|
||||
"print(B)\n",
|
||||
"# Compare our own algorithm with pinv\n",
|
||||
"C = SVDinv(A)\n",
|
||||
"print(np.abs(C-B))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by **Numpy**.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Ridge and LASSO Regression\n",
|
||||
"\n",
|
||||
"Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n",
|
||||
|
||||
@@ -34,7 +34,7 @@ We used the SVD to analyse the matrix to invert in ordinary lineat regression
|
||||
Since the matrices here have dimension $p\times p$, with $p$ corresponding to the singular values, we defined last week the matrix
|
||||
!bt
|
||||
\[
|
||||
\bm{\Sigma}^T\bm{\Sigma} = \begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\\ \end{bmatrix},
|
||||
\bm{\Sigma}^T\bm{\Sigma} = \begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\end{bmatrix},
|
||||
\]
|
||||
!et
|
||||
where the tilde-matrix $\tilde{\bm{\Sigma}}$ is a matrix of dimension $p\times p$ containing only the singular values $\sigma_i$, that is
|
||||
@@ -178,15 +178,64 @@ def SVDinv(A):
|
||||
|
||||
|
||||
#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])
|
||||
X = np.array( [ [1,2],[2,3]])
|
||||
# Non-singular square matrix
|
||||
X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])
|
||||
print(X)
|
||||
A = np.transpose(X) @ X
|
||||
# Brute force inversion
|
||||
B = np.linalg.inv(A)
|
||||
B = np.linalg.inv(A) # here we could use np.linalg.pinv(A)
|
||||
C = SVDinv(A)
|
||||
print(np.abs(B-C))
|
||||
|
||||
!ec
|
||||
|
||||
!split
|
||||
===== Inverse of Rectangular Matrix =====
|
||||
|
||||
Although our matrix to invert $\bm{X}^T\bm{X}$ is a square matrix, our matrix may be singular.
|
||||
|
||||
The pseudoinverse is the generalization of the matrix inverse for square matrices to
|
||||
rectangular matrices where the number of rows and columns are not equal.
|
||||
|
||||
It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.
|
||||
It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.
|
||||
|
||||
Using the SVD we can obtain the pseudoinverse of a matrix $\bm{A}$ (labeled here as $\bm{A}_{\mathrm{PI}}$
|
||||
!bt
|
||||
\[
|
||||
\bm{A}_{\mathrm{PI}}= \bm{V}\bm{D}_{\mathrm{PI}}\bm{U}^T,
|
||||
\]
|
||||
!et
|
||||
where $\bm{D}_{\mathrm{PI}}$ can be calculated by creating a diagonal matrix from $\bm{Sigma}$ where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.
|
||||
|
||||
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
# SVD inversion
|
||||
def SVDinv(A):
|
||||
U, s, VT = np.linalg.svd(A)
|
||||
# reciprocals of singular values of s
|
||||
d = 1.0 / s
|
||||
# create m x n D matrix
|
||||
D = np.zeros(A.shape)
|
||||
# populate D with n x n diagonal matrix
|
||||
D[:A.shape[1], :A.shape[1]] = np.diag(d)
|
||||
UT = np.transpose(U)
|
||||
V = np.transpose(VT)
|
||||
return np.matmul(V,np.matmul(D.T,UT))
|
||||
|
||||
|
||||
A = np.array([ [0.3, 0.4], [0.5, 0.6], [0.7, 0.8],[0.9, 1.0]])
|
||||
print(A)
|
||||
# Brute force inversion of super-collinear matrix
|
||||
B = np.linalg.pinv(A)
|
||||
print(B)
|
||||
# Compare our own algorithm with pinv
|
||||
C = SVDinv(A)
|
||||
print(np.abs(C-B))
|
||||
|
||||
!ec
|
||||
As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by _Numpy_.
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user