lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
mhjensen
2020-08-19 11:52:34 +02:00
parent 918b257206
commit e58831a92f
136 changed files with 28678 additions and 28400 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Regression/html/._Regression-bs106.html
+270 -314
View File
@@ -41,24 +41,21 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('Why Linear Regression (aka Ordinary Least Squares and family)',
'sections': [('To do list', 2, None, '___sec0'),
('Why Linear Regression (aka Ordinary Least Squares and family)',
2,
None,
'___sec0'),
('Regression analysis, overarching aims', 2, None, '___sec1'),
('Regression analysis, overarching aims II', 2, None, '___sec2'),
('Examples', 2, None, '___sec3'),
('General linear models', 2, None, '___sec4'),
'___sec1'),
('Regression analysis, overarching aims', 2, None, '___sec2'),
('Regression analysis, overarching aims II', 2, None, '___sec3'),
('Examples', 2, None, '___sec4'),
('General linear models', 2, None, '___sec5'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'___sec5'),
'___sec6'),
('Rewriting the fitting procedure as a linear algebra problem, '
'more details',
2,
None,
'___sec6'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec7'),
@@ -66,184 +63,188 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Optimizing our parameters', 2, None, '___sec9'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'___sec9'),
('Optimizing our parameters', 2, None, '___sec10'),
('Our model for the nuclear binding energies',
2,
None,
'___sec10'),
('Optimizing our parameters, more details', 2, None, '___sec11'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec12'),
'___sec11'),
('Optimizing our parameters, more details', 2, None, '___sec12'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec13'),
('Some useful matrix and vector expressions',
2,
None,
'___sec14'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec14'),
('Some useful matrix and vector expressions',
2,
None,
'___sec15'),
('Own code for Ordinary Least Squares', 2, None, '___sec16'),
('Interpretations and optimizing our parameters',
2,
None,
'___sec16'),
('Own code for Ordinary Least Squares', 2, None, '___sec17'),
('Adding error analysis and training set up',
2,
None,
'___sec17'),
('The $\\chi^2$ function', 2, None, '___sec18'),
'___sec18'),
('The $\\chi^2$ function', 2, None, '___sec19'),
('The $\\chi^2$ function', 2, None, '___sec20'),
('The $\\chi^2$ function', 2, None, '___sec21'),
('The $\\chi^2$ function', 2, None, '___sec22'),
('The $\\chi^2$ function', 2, None, '___sec23'),
('The $\\chi^2$ function', 2, None, '___sec24'),
('Fitting an Equation of State for Dense Nuclear Matter',
2,
None,
'___sec24'),
('The code', 2, None, '___sec25'),
'___sec25'),
('The code', 2, None, '___sec26'),
('Splitting our Data in Training and Test data',
2,
None,
'___sec26'),
('The Boston housing data example', 2, None, '___sec27'),
('Housing data, the code', 2, None, '___sec28'),
('The singular value decomposition', 2, None, '___sec29'),
('Linear Regression Problems', 2, None, '___sec30'),
('Fixing the singularity', 2, None, '___sec31'),
('Basic math of the SVD', 2, None, '___sec32'),
('The SVD, a Fantastic Algorithm', 2, None, '___sec33'),
('Another Example', 2, None, '___sec34'),
('Economy-size SVD', 2, None, '___sec35'),
('Mathematical Properties', 2, None, '___sec36'),
('Ridge and LASSO Regression', 2, None, '___sec37'),
('More on Ridge Regression', 2, None, '___sec38'),
('Interpreting the Ridge results', 2, None, '___sec39'),
('More interpretations', 2, None, '___sec40'),
('Codes for the SVD', 2, None, '___sec41'),
('A better understanding of regularization', 2, None, '___sec42'),
'___sec27'),
('The Boston housing data example', 2, None, '___sec28'),
('Housing data, the code', 2, None, '___sec29'),
('The singular value decomposition', 2, None, '___sec30'),
('Linear Regression Problems', 2, None, '___sec31'),
('Fixing the singularity', 2, None, '___sec32'),
('Basic math of the SVD', 2, None, '___sec33'),
('The SVD, a Fantastic Algorithm', 2, None, '___sec34'),
('Another Example', 2, None, '___sec35'),
('Economy-size SVD', 2, None, '___sec36'),
('Mathematical Properties', 2, None, '___sec37'),
('Ridge and LASSO Regression', 2, None, '___sec38'),
('More on Ridge Regression', 2, None, '___sec39'),
('Interpreting the Ridge results', 2, None, '___sec40'),
('More interpretations', 2, None, '___sec41'),
('Codes for the SVD', 2, None, '___sec42'),
('A better understanding of regularization', 2, None, '___sec43'),
('Decomposing the OLS and Ridge expressions',
2,
None,
'___sec43'),
('Spectral Decomposition of the OLS', 2, None, '___sec44'),
('Where are we going?', 2, None, '___sec45'),
('Resampling methods', 2, None, '___sec46'),
'___sec44'),
('Spectral Decomposition of the OLS', 2, None, '___sec45'),
('Where are we going?', 2, None, '___sec46'),
('Resampling methods', 2, None, '___sec47'),
('Resampling approaches can be computationally expensive',
2,
None,
'___sec47'),
('Why resampling methods ?', 2, None, '___sec48'),
('Statistical analysis', 2, None, '___sec49'),
('Statistics', 2, None, '___sec50'),
('Statistics, moments', 2, None, '___sec51'),
('Statistics, central moments', 2, None, '___sec52'),
('Statistics, covariance', 2, None, '___sec53'),
('Statistics, more covariance', 2, None, '___sec54'),
('Covariance example', 2, None, '___sec55'),
('Covariance in numpy', 2, None, '___sec56'),
('Statistics, independent variables', 2, None, '___sec57'),
('Statistics, more variance', 2, None, '___sec58'),
('Statistics and stochastic processes', 2, None, '___sec59'),
('Statistics and sample variables', 2, None, '___sec60'),
'___sec48'),
('Why resampling methods ?', 2, None, '___sec49'),
('Statistical analysis', 2, None, '___sec50'),
('Statistics', 2, None, '___sec51'),
('Statistics, moments', 2, None, '___sec52'),
('Statistics, central moments', 2, None, '___sec53'),
('Statistics, covariance', 2, None, '___sec54'),
('Statistics, more covariance', 2, None, '___sec55'),
('Covariance example', 2, None, '___sec56'),
('Covariance in numpy', 2, None, '___sec57'),
('Statistics, independent variables', 2, None, '___sec58'),
('Statistics, more variance', 2, None, '___sec59'),
('Statistics and stochastic processes', 2, None, '___sec60'),
('Statistics and sample variables', 2, None, '___sec61'),
('Statistics, sample variance and covariance',
2,
None,
'___sec61'),
('Statistics, law of large numbers', 2, None, '___sec62'),
('Statistics, more on sample error', 2, None, '___sec63'),
('Statistics', 2, None, '___sec64'),
('Statistics, central limit theorem', 2, None, '___sec65'),
('Statistics, more technicalities', 2, None, '___sec66'),
('Statistics', 2, None, '___sec67'),
('Statistics and sample variance', 2, None, '___sec68'),
('Statistics, uncorrelated results', 2, None, '___sec69'),
('Statistics, computations', 2, None, '___sec70'),
'___sec62'),
('Statistics, law of large numbers', 2, None, '___sec63'),
('Statistics, more on sample error', 2, None, '___sec64'),
('Statistics', 2, None, '___sec65'),
('Statistics, central limit theorem', 2, None, '___sec66'),
('Statistics, more technicalities', 2, None, '___sec67'),
('Statistics', 2, None, '___sec68'),
('Statistics and sample variance', 2, None, '___sec69'),
('Statistics, uncorrelated results', 2, None, '___sec70'),
('Statistics, computations', 2, None, '___sec71'),
('Statistics, more on computations of errors',
2,
None,
'___sec71'),
('Statistics, wrapping up 1', 2, None, '___sec72'),
('Statistics, final expression', 2, None, '___sec73'),
'___sec72'),
('Statistics, wrapping up 1', 2, None, '___sec73'),
('Statistics, final expression', 2, None, '___sec74'),
('Statistics, effective number of correlations',
2,
None,
'___sec74'),
'___sec75'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
None,
'___sec75'),
('Assumptions made', 2, None, '___sec76'),
('Expectation value and variance', 2, None, '___sec77'),
'___sec76'),
('Assumptions made', 2, None, '___sec77'),
('Expectation value and variance', 2, None, '___sec78'),
('Expectation value and variance for $\\boldsymbol{\\beta}$',
2,
None,
'___sec78'),
('Resampling methods', 2, None, '___sec79'),
'___sec79'),
('Resampling methods', 2, None, '___sec80'),
('Resampling methods: Jackknife and Bootstrap',
2,
None,
'___sec80'),
('Resampling methods: Jackknife', 2, None, '___sec81'),
('Jackknife code example', 2, None, '___sec82'),
('Resampling methods: Bootstrap', 2, None, '___sec83'),
('Resampling methods: Bootstrap background', 2, None, '___sec84'),
'___sec81'),
('Resampling methods: Jackknife', 2, None, '___sec82'),
('Jackknife code example', 2, None, '___sec83'),
('Resampling methods: Bootstrap', 2, None, '___sec84'),
('Resampling methods: Bootstrap background', 2, None, '___sec85'),
('Resampling methods: More Bootstrap background',
2,
None,
'___sec85'),
('Resampling methods: Bootstrap approach', 2, None, '___sec86'),
('Resampling methods: Bootstrap steps', 2, None, '___sec87'),
('Code example for the Bootstrap method', 2, None, '___sec88'),
('Various steps in cross-validation', 2, None, '___sec89'),
'___sec86'),
('Resampling methods: Bootstrap approach', 2, None, '___sec87'),
('Resampling methods: Bootstrap steps', 2, None, '___sec88'),
('Code example for the Bootstrap method', 2, None, '___sec89'),
('Various steps in cross-validation', 2, None, '___sec90'),
('How to set up the cross-validation for Ridge and/or Lasso',
2,
None,
'___sec90'),
('Cross-validation in brief', 2, None, '___sec91'),
'___sec91'),
('Cross-validation in brief', 2, None, '___sec92'),
('Code Example for Cross-validation and $k$-fold '
'Cross-validation',
2,
None,
'___sec92'),
('The bias-variance tradeoff', 2, None, '___sec93'),
('Example code for Bias-Variance tradeoff', 2, None, '___sec94'),
('Understanding what happens', 2, None, '___sec95'),
('Summing up', 2, None, '___sec96'),
'___sec93'),
('The bias-variance tradeoff', 2, None, '___sec94'),
('Example code for Bias-Variance tradeoff', 2, None, '___sec95'),
('Understanding what happens', 2, None, '___sec96'),
('Summing up', 2, None, '___sec97'),
("Another Example from Scikit-Learn's Repository",
2,
None,
'___sec97'),
'___sec98'),
('More examples on bootstrap and cross-validation and errors',
2,
None,
'___sec98'),
'___sec99'),
('The same example but now with cross-validation',
2,
None,
'___sec99'),
('Cross-validation with Ridge', 2, None, '___sec100'),
('The Ising model', 2, None, '___sec101'),
'___sec100'),
('Cross-validation with Ridge', 2, None, '___sec101'),
('The Ising model', 2, None, '___sec102'),
('Reformulating the problem to suit regression',
2,
None,
'___sec102'),
('Linear regression', 2, None, '___sec103'),
('Singular Value decomposition', 2, None, '___sec104'),
('The one-dimensional Ising model', 2, None, '___sec105'),
('Ridge regression', 2, None, '___sec106'),
('LASSO regression', 2, None, '___sec107'),
'___sec103'),
('Linear regression', 2, None, '___sec104'),
('Singular Value decomposition', 2, None, '___sec105'),
('The one-dimensional Ising model', 2, None, '___sec106'),
('Ridge regression', 2, None, '___sec107'),
('LASSO regression', 2, None, '___sec108'),
('Performance as function of the regularization parameter',
2,
None,
'___sec108'),
'___sec109'),
('Finding the optimal value of $\\lambda$',
2,
None,
'___sec109')]}
'___sec110')]}
end of tocinfo -->
<body>
@@ -281,116 +282,117 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">Why Linear Regression (aka Ordinary Least Squares and family)</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Regression analysis, overarching aims II</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Examples</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, more details</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs001.html#___sec0" style="font-size: 80%;">To do list</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs002.html#___sec1" style="font-size: 80%;">Why Linear Regression (aka Ordinary Least Squares and family)</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs003.html#___sec2" style="font-size: 80%;">Regression analysis, overarching aims</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs004.html#___sec3" style="font-size: 80%;">Regression analysis, overarching aims II</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs005.html#___sec4" style="font-size: 80%;">Examples</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs006.html#___sec5" style="font-size: 80%;">General linear models</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs007.html#___sec6" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs008.html#___sec7" style="font-size: 80%;">Rewriting the fitting procedure as a linear algebra problem, more details</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs009.html#___sec8" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Our model for the nuclear binding energies</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs013.html#___sec12" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs010.html#___sec9" style="font-size: 80%;">Generalizing the fitting procedure as a linear algebra problem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs011.html#___sec10" style="font-size: 80%;">Optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs012.html#___sec11" style="font-size: 80%;">Our model for the nuclear binding energies</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs013.html#___sec12" style="font-size: 80%;">Optimizing our parameters, more details</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs014.html#___sec13" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs015.html#___sec14" style="font-size: 80%;">Some useful matrix and vector expressions</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs016.html#___sec15" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs017.html#___sec16" style="font-size: 80%;">Own code for Ordinary Least Squares</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs018.html#___sec17" style="font-size: 80%;">Adding error analysis and training set up</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs019.html#___sec18" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs015.html#___sec14" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs016.html#___sec15" style="font-size: 80%;">Some useful matrix and vector expressions</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs017.html#___sec16" style="font-size: 80%;">Interpretations and optimizing our parameters</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs018.html#___sec17" style="font-size: 80%;">Own code for Ordinary Least Squares</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs019.html#___sec18" style="font-size: 80%;">Adding error analysis and training set up</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs020.html#___sec19" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs021.html#___sec20" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs022.html#___sec21" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs023.html#___sec22" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs024.html#___sec23" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs025.html#___sec24" style="font-size: 80%;">Fitting an Equation of State for Dense Nuclear Matter</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs026.html#___sec25" style="font-size: 80%;">The code</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs027.html#___sec26" style="font-size: 80%;">Splitting our Data in Training and Test data</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs028.html#___sec27" style="font-size: 80%;">The Boston housing data example</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs029.html#___sec28" style="font-size: 80%;">Housing data, the code</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs030.html#___sec29" style="font-size: 80%;">The singular value decomposition</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs031.html#___sec30" style="font-size: 80%;">Linear Regression Problems</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs032.html#___sec31" style="font-size: 80%;">Fixing the singularity</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs033.html#___sec32" style="font-size: 80%;">Basic math of the SVD</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs034.html#___sec33" style="font-size: 80%;">The SVD, a Fantastic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs035.html#___sec34" style="font-size: 80%;">Another Example</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs036.html#___sec35" style="font-size: 80%;">Economy-size SVD</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs037.html#___sec36" style="font-size: 80%;">Mathematical Properties</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs038.html#___sec37" style="font-size: 80%;">Ridge and LASSO Regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs039.html#___sec38" style="font-size: 80%;">More on Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs040.html#___sec39" style="font-size: 80%;">Interpreting the Ridge results</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs041.html#___sec40" style="font-size: 80%;">More interpretations</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs042.html#___sec41" style="font-size: 80%;">Codes for the SVD</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs043.html#___sec42" style="font-size: 80%;">A better understanding of regularization</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs044.html#___sec43" style="font-size: 80%;">Decomposing the OLS and Ridge expressions</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs045.html#___sec44" style="font-size: 80%;">Spectral Decomposition of the OLS</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs046.html#___sec45" style="font-size: 80%;">Where are we going?</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs047.html#___sec46" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs048.html#___sec47" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs049.html#___sec48" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs050.html#___sec49" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs051.html#___sec50" style="font-size: 80%;">Statistics</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs052.html#___sec51" style="font-size: 80%;">Statistics, moments</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs053.html#___sec52" style="font-size: 80%;">Statistics, central moments</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs054.html#___sec53" style="font-size: 80%;">Statistics, covariance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs055.html#___sec54" style="font-size: 80%;">Statistics, more covariance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs056.html#___sec55" style="font-size: 80%;">Covariance example</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs057.html#___sec56" style="font-size: 80%;">Covariance in numpy</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs058.html#___sec57" style="font-size: 80%;">Statistics, independent variables</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs059.html#___sec58" style="font-size: 80%;">Statistics, more variance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs060.html#___sec59" style="font-size: 80%;">Statistics and stochastic processes</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs061.html#___sec60" style="font-size: 80%;">Statistics and sample variables</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs062.html#___sec61" style="font-size: 80%;">Statistics, sample variance and covariance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs063.html#___sec62" style="font-size: 80%;">Statistics, law of large numbers</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs064.html#___sec63" style="font-size: 80%;">Statistics, more on sample error</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs065.html#___sec64" style="font-size: 80%;">Statistics</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs066.html#___sec65" style="font-size: 80%;">Statistics, central limit theorem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs067.html#___sec66" style="font-size: 80%;">Statistics, more technicalities</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs068.html#___sec67" style="font-size: 80%;">Statistics</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs069.html#___sec68" style="font-size: 80%;">Statistics and sample variance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs070.html#___sec69" style="font-size: 80%;">Statistics, uncorrelated results</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs071.html#___sec70" style="font-size: 80%;">Statistics, computations</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs072.html#___sec71" style="font-size: 80%;">Statistics, more on computations of errors</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs073.html#___sec72" style="font-size: 80%;">Statistics, wrapping up 1</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs074.html#___sec73" style="font-size: 80%;">Statistics, final expression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs075.html#___sec74" style="font-size: 80%;">Statistics, effective number of correlations</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs076.html#___sec75" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs077.html#___sec76" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs078.html#___sec77" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs079.html#___sec78" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs080.html#___sec79" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs081.html#___sec80" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs082.html#___sec81" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs083.html#___sec82" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs084.html#___sec83" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs085.html#___sec84" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs086.html#___sec85" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs087.html#___sec86" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs088.html#___sec87" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs089.html#___sec88" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs090.html#___sec89" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs091.html#___sec90" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs092.html#___sec91" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs093.html#___sec92" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs094.html#___sec93" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs095.html#___sec94" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs096.html#___sec95" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs097.html#___sec96" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs098.html#___sec97" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs099.html#___sec98" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs100.html#___sec99" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs101.html#___sec100" style="font-size: 80%;">Cross-validation with Ridge</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs102.html#___sec101" style="font-size: 80%;">The Ising model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs103.html#___sec102" style="font-size: 80%;">Reformulating the problem to suit regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs104.html#___sec103" style="font-size: 80%;">Linear regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs105.html#___sec104" style="font-size: 80%;">Singular Value decomposition</a></li>
<!-- navigation toc: --> <li><a href="#___sec105" style="font-size: 80%;">The one-dimensional Ising model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs107.html#___sec106" style="font-size: 80%;">Ridge regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs108.html#___sec107" style="font-size: 80%;">LASSO regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs109.html#___sec108" style="font-size: 80%;">Performance as function of the regularization parameter</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs110.html#___sec109" style="font-size: 80%;">Finding the optimal value of \( \lambda \)</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs025.html#___sec24" style="font-size: 80%;">The \( \chi^2 \) function</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs026.html#___sec25" style="font-size: 80%;">Fitting an Equation of State for Dense Nuclear Matter</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs027.html#___sec26" style="font-size: 80%;">The code</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs028.html#___sec27" style="font-size: 80%;">Splitting our Data in Training and Test data</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs029.html#___sec28" style="font-size: 80%;">The Boston housing data example</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs030.html#___sec29" style="font-size: 80%;">Housing data, the code</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs031.html#___sec30" style="font-size: 80%;">The singular value decomposition</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs032.html#___sec31" style="font-size: 80%;">Linear Regression Problems</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs033.html#___sec32" style="font-size: 80%;">Fixing the singularity</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs034.html#___sec33" style="font-size: 80%;">Basic math of the SVD</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs035.html#___sec34" style="font-size: 80%;">The SVD, a Fantastic Algorithm</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs036.html#___sec35" style="font-size: 80%;">Another Example</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs037.html#___sec36" style="font-size: 80%;">Economy-size SVD</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs038.html#___sec37" style="font-size: 80%;">Mathematical Properties</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs039.html#___sec38" style="font-size: 80%;">Ridge and LASSO Regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs040.html#___sec39" style="font-size: 80%;">More on Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs041.html#___sec40" style="font-size: 80%;">Interpreting the Ridge results</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs042.html#___sec41" style="font-size: 80%;">More interpretations</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs043.html#___sec42" style="font-size: 80%;">Codes for the SVD</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs044.html#___sec43" style="font-size: 80%;">A better understanding of regularization</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs045.html#___sec44" style="font-size: 80%;">Decomposing the OLS and Ridge expressions</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs046.html#___sec45" style="font-size: 80%;">Spectral Decomposition of the OLS</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs047.html#___sec46" style="font-size: 80%;">Where are we going?</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs048.html#___sec47" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs049.html#___sec48" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs050.html#___sec49" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs051.html#___sec50" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs052.html#___sec51" style="font-size: 80%;">Statistics</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs053.html#___sec52" style="font-size: 80%;">Statistics, moments</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs054.html#___sec53" style="font-size: 80%;">Statistics, central moments</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs055.html#___sec54" style="font-size: 80%;">Statistics, covariance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs056.html#___sec55" style="font-size: 80%;">Statistics, more covariance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs057.html#___sec56" style="font-size: 80%;">Covariance example</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs058.html#___sec57" style="font-size: 80%;">Covariance in numpy</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs059.html#___sec58" style="font-size: 80%;">Statistics, independent variables</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs060.html#___sec59" style="font-size: 80%;">Statistics, more variance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs061.html#___sec60" style="font-size: 80%;">Statistics and stochastic processes</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs062.html#___sec61" style="font-size: 80%;">Statistics and sample variables</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs063.html#___sec62" style="font-size: 80%;">Statistics, sample variance and covariance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs064.html#___sec63" style="font-size: 80%;">Statistics, law of large numbers</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs065.html#___sec64" style="font-size: 80%;">Statistics, more on sample error</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs066.html#___sec65" style="font-size: 80%;">Statistics</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs067.html#___sec66" style="font-size: 80%;">Statistics, central limit theorem</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs068.html#___sec67" style="font-size: 80%;">Statistics, more technicalities</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs069.html#___sec68" style="font-size: 80%;">Statistics</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs070.html#___sec69" style="font-size: 80%;">Statistics and sample variance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs071.html#___sec70" style="font-size: 80%;">Statistics, uncorrelated results</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs072.html#___sec71" style="font-size: 80%;">Statistics, computations</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs073.html#___sec72" style="font-size: 80%;">Statistics, more on computations of errors</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs074.html#___sec73" style="font-size: 80%;">Statistics, wrapping up 1</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs075.html#___sec74" style="font-size: 80%;">Statistics, final expression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs076.html#___sec75" style="font-size: 80%;">Statistics, effective number of correlations</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs077.html#___sec76" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs078.html#___sec77" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs079.html#___sec78" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs080.html#___sec79" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs081.html#___sec80" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs082.html#___sec81" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs083.html#___sec82" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs084.html#___sec83" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs085.html#___sec84" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs086.html#___sec85" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs087.html#___sec86" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs088.html#___sec87" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs089.html#___sec88" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs090.html#___sec89" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs091.html#___sec90" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs092.html#___sec91" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs093.html#___sec92" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs094.html#___sec93" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs095.html#___sec94" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs096.html#___sec95" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs097.html#___sec96" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs098.html#___sec97" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs099.html#___sec98" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs100.html#___sec99" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs101.html#___sec100" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs102.html#___sec101" style="font-size: 80%;">Cross-validation with Ridge</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs103.html#___sec102" style="font-size: 80%;">The Ising model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs104.html#___sec103" style="font-size: 80%;">Reformulating the problem to suit regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs105.html#___sec104" style="font-size: 80%;">Linear regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec105" style="font-size: 80%;">Singular Value decomposition</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs107.html#___sec106" style="font-size: 80%;">The one-dimensional Ising model</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs108.html#___sec107" style="font-size: 80%;">Ridge regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs109.html#___sec108" style="font-size: 80%;">LASSO regression</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs110.html#___sec109" style="font-size: 80%;">Performance as function of the regularization parameter</a></li>
<!-- navigation toc: --> <li><a href="._Regression-bs111.html#___sec110" style="font-size: 80%;">Finding the optimal value of \( \lambda \)</a></li>
</ul>
</li>
@@ -406,128 +408,74 @@ MathJax.Hub.Config({
<a name="part0106"></a>
<!-- !split -->
<h2 id="___sec105" class="anchor">The one-dimensional Ising model </h2>
<h2 id="___sec105" class="anchor">Singular Value decomposition </h2>
<p>
Let us bring back the Ising model again, but now with an additional
focus on Ridge and Lasso regression as well. We repeat some of the
basic parts of the Ising model and the setup of the training and test
data. The one-dimensional Ising model with nearest neighbor
interaction, no external field and a constant coupling constant \( J \) is
given by
Doing the inversion directly turns out to be a bad idea since the matrix
\( \boldsymbol{X}^T\boldsymbol{X} \) is singular. An alternative approach is to use the <b>singular
value decomposition</b>. Using the definition of the Moore-Penrose
pseudoinverse we can write the equation for \( \boldsymbol{\beta} \) as
$$
\boldsymbol{\beta} = \boldsymbol{X}^{+}\boldsymbol{y},
$$
<p>
where the pseudoinverse of \( \boldsymbol{X} \) is given by
$$
\boldsymbol{X}^{+} = \frac{\boldsymbol{X}^T}{\boldsymbol{X}^T\boldsymbol{X}}.
$$
<p>
Using singular value decomposition we can decompose the matrix \( \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma} \boldsymbol{V}^T \),
where \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal(unitary) matrices and \( \boldsymbol{\Sigma} \) contains the singular values (more details below).
where \( X^{+} = V\Sigma^{+} U^T \). This reduces the equation for
\( \omega \) to
$$
\begin{align}
H = -J \sum_{k}^L s_k s_{k + 1},
\tag{27}
\end{align}
$$
where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins in the system is determined by \( L \). For the one-dimensional system there is no phase transition.
<p>
We will look at a system of \( L = 40 \) spins with a coupling constant of \( J = 1 \). To get enough training data we will generate 10000 states with their respective energies.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.axes_grid1</span> <span style="color: #008000; font-weight: bold">import</span> make_axes_locatable
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">seaborn</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sns</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scipy.linalg</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">scl</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skl</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">tqdm</span>
sns<span style="color: #666666">.</span>set(color_codes<span style="color: #666666">=</span><span style="color: #008000">True</span>)
cmap_args<span style="color: #666666">=</span><span style="color: #008000">dict</span>(vmin<span style="color: #666666">=-1.</span>, vmax<span style="color: #666666">=1.</span>, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&#39;seismic&#39;</span>)
L <span style="color: #666666">=</span> <span style="color: #666666">40</span>
n <span style="color: #666666">=</span> <span style="color: #008000">int</span>(<span style="color: #666666">1e4</span>)
spins <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>choice([<span style="color: #666666">-1</span>, <span style="color: #666666">1</span>], size<span style="color: #666666">=</span>(n, L))
J <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
energies <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(n)
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(n):
energies[i] <span style="color: #666666">=</span> <span style="color: #666666">-</span> J <span style="color: #666666">*</span> np<span style="color: #666666">.</span>dot(spins[i], np<span style="color: #666666">.</span>roll(spins[i], <span style="color: #666666">1</span>))
</pre></div>
<p>
A more general form for the one-dimensional Ising model is
$$
\begin{align}
H = - \sum_j^L \sum_k^L s_j s_k J_{jk}.
\tag{28}
\boldsymbol{\beta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}.
\tag{26}
\end{align}
$$
<p>
Here we allow for interactions beyond the nearest neighbors and a more
adaptive coupling matrix. This latter expression can be formulated as
a matrix-product on the form
$$
\begin{align}
H = X J,
\tag{29}
\end{align}
$$
<p>
where \( X_{jk} = s_j s_k \) and \( J \) is the matrix consisting of the
elements \( -J_{jk} \). This form of writing the energy fits perfectly
with the form utilized in linear regression, viz.
$$
\begin{align}
\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}.
\tag{30}
\end{align}
$$
We organize the data as we did above
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((n, L <span style="color: #666666">**</span> <span style="color: #666666">2</span>))
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(n):
X[i] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>outer(spins[i], spins[i])<span style="color: #666666">.</span>ravel()
y <span style="color: #666666">=</span> energies
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(X, y, test_size<span style="color: #666666">=0.96</span>)
X_train_own <span style="color: #666666">=</span> np<span style="color: #666666">.</span>concatenate(
(np<span style="color: #666666">.</span>ones(<span style="color: #008000">len</span>(X_train))[:, np<span style="color: #666666">.</span>newaxis], X_train),
axis<span style="color: #666666">=1</span>
)
X_test_own <span style="color: #666666">=</span> np<span style="color: #666666">.</span>concatenate(
(np<span style="color: #666666">.</span>ones(<span style="color: #008000">len</span>(X_test))[:, np<span style="color: #666666">.</span>newaxis], X_test),
axis<span style="color: #666666">=1</span>
)
</pre></div>
<p>
We will do all fitting with <b>Scikit-Learn</b>,
Note that solving this equation by actually doing the pseudoinverse
(which is what we will do) is not a good idea as this operation scales
as \( \mathcal{O}(n^3) \), where \( n \) is the number of elements in a
general matrix. Instead, doing \( QR \)-factorization and solving the
linear system as an equation would reduce this down to
\( \mathcal{O}(n^2) \) operations.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>clf <span style="color: #666666">=</span> skl<span style="color: #666666">.</span>LinearRegression()<span style="color: #666666">.</span>fit(X_train, y_train)
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">ols_svd</span>(x: np<span style="color: #666666">.</span>ndarray, y: np<span style="color: #666666">.</span>ndarray) <span style="color: #666666">-&gt;</span> np<span style="color: #666666">.</span>ndarray:
u, s, v <span style="color: #666666">=</span> scl<span style="color: #666666">.</span>svd(x)
<span style="color: #008000; font-weight: bold">return</span> v<span style="color: #666666">.</span>T <span style="color: #666666">@</span> scl<span style="color: #666666">.</span>pinv(scl<span style="color: #666666">.</span>diagsvd(s, u<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>], v<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>])) <span style="color: #666666">@</span> u<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y
</pre></div>
<p>
When extracting the \( J \)-matrix we make sure to remove the intercept
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>J_sk <span style="color: #666666">=</span> clf<span style="color: #666666">.</span>coef_<span style="color: #666666">.</span>reshape(L, L)
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>beta <span style="color: #666666">=</span> ols_svd(X_train_own,y_train)
</pre></div>
<p>
And then we plot the results
When extracting the \( J \)-matrix we need to make sure that we remove the intercept, as is done here
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>J <span style="color: #666666">=</span> beta[<span style="color: #666666">1</span>:]<span style="color: #666666">.</span>reshape(L, L)
</pre></div>
<p>
A way of looking at the coefficients in \( J \) is to plot the matrices as images.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">20</span>, <span style="color: #666666">14</span>))
im <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>imshow(J_sk, <span style="color: #666666">**</span>cmap_args)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;LinearRegression from Scikit-learn&quot;</span>, fontsize<span style="color: #666666">=18</span>)
im <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>imshow(J, <span style="color: #666666">**</span>cmap_args)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;OLS&quot;</span>, fontsize<span style="color: #666666">=18</span>)
plt<span style="color: #666666">.</span>xticks(fontsize<span style="color: #666666">=18</span>)
plt<span style="color: #666666">.</span>yticks(fontsize<span style="color: #666666">=18</span>)
cb <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>colorbar(im)
@@ -535,7 +483,14 @@ cb<span style="color: #666666">.</span>ax<span style="color: #666666">.</span>se
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
The results perfectly with our previous discussion where we used our own code.
It is interesting to note that OLS
considers both \( J_{j, j + 1} = -0.5 \) and \( J_{j, j - 1} = -0.5 \) as
valid matrix elements for \( J \).
In our discussion below on hyperparameters and Ridge and Lasso regression we will see that
this problem can be removed, partly and only with Lasso regression.
<p>
In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD?
<p>
<p>
@@ -557,6 +512,7 @@ The results perfectly with our previous discussion where we used our own code.
<li><a href="._Regression-bs108.html">109</a></li>
<li><a href="._Regression-bs109.html">110</a></li>
<li><a href="._Regression-bs110.html">111</a></li>
<li><a href="._Regression-bs111.html">112</a></li>
<li><a href="._Regression-bs107.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->