From 3fd6a57c6e1cee7a91028584f39f25cc4e12a6d5 Mon Sep 17 00:00:00 2001 From: mhjensen Date: Fri, 30 Aug 2019 05:47:54 +0200 Subject: [PATCH] Fixed type and shifted Ising model to end for Regression slides --- .../Regression/html/._Regression-bs000.html | 280 +++--- .../Regression/html/._Regression-bs001.html | 278 +++--- .../Regression/html/._Regression-bs002.html | 278 +++--- .../Regression/html/._Regression-bs003.html | 278 +++--- .../Regression/html/._Regression-bs004.html | 278 +++--- .../Regression/html/._Regression-bs005.html | 278 +++--- .../Regression/html/._Regression-bs006.html | 278 +++--- .../Regression/html/._Regression-bs007.html | 278 +++--- .../Regression/html/._Regression-bs008.html | 278 +++--- .../Regression/html/._Regression-bs009.html | 278 +++--- .../Regression/html/._Regression-bs010.html | 278 +++--- .../Regression/html/._Regression-bs011.html | 278 +++--- .../Regression/html/._Regression-bs012.html | 278 +++--- .../Regression/html/._Regression-bs013.html | 278 +++--- .../Regression/html/._Regression-bs014.html | 278 +++--- .../Regression/html/._Regression-bs015.html | 278 +++--- .../Regression/html/._Regression-bs016.html | 278 +++--- .../Regression/html/._Regression-bs017.html | 278 +++--- .../Regression/html/._Regression-bs018.html | 278 +++--- .../Regression/html/._Regression-bs019.html | 278 +++--- .../Regression/html/._Regression-bs020.html | 278 +++--- .../Regression/html/._Regression-bs021.html | 278 +++--- .../Regression/html/._Regression-bs022.html | 278 +++--- .../Regression/html/._Regression-bs023.html | 278 +++--- .../Regression/html/._Regression-bs024.html | 278 +++--- .../Regression/html/._Regression-bs025.html | 278 +++--- .../Regression/html/._Regression-bs026.html | 278 +++--- .../Regression/html/._Regression-bs027.html | 278 +++--- .../Regression/html/._Regression-bs028.html | 278 +++--- .../Regression/html/._Regression-bs029.html | 278 +++--- .../Regression/html/._Regression-bs030.html | 284 +++--- .../Regression/html/._Regression-bs031.html | 360 ++++--- .../Regression/html/._Regression-bs032.html | 324 +++---- .../Regression/html/._Regression-bs033.html | 333 ++++--- .../Regression/html/._Regression-bs034.html | 366 +++---- .../Regression/html/._Regression-bs035.html | 341 ++++--- .../Regression/html/._Regression-bs036.html | 313 +++--- .../Regression/html/._Regression-bs037.html | 347 +++---- .../Regression/html/._Regression-bs038.html | 337 ++++--- .../Regression/html/._Regression-bs039.html | 341 +++---- .../Regression/html/._Regression-bs040.html | 307 +++--- .../Regression/html/._Regression-bs041.html | 325 +++---- .../Regression/html/._Regression-bs042.html | 337 +++---- .../Regression/html/._Regression-bs043.html | 348 +++---- .../Regression/html/._Regression-bs044.html | 314 +++--- .../Regression/html/._Regression-bs045.html | 320 +++---- .../Regression/html/._Regression-bs046.html | 304 +++--- .../Regression/html/._Regression-bs047.html | 311 +++--- .../Regression/html/._Regression-bs048.html | 310 +++--- .../Regression/html/._Regression-bs049.html | 312 +++--- .../Regression/html/._Regression-bs050.html | 311 +++--- .../Regression/html/._Regression-bs051.html | 319 +++--- .../Regression/html/._Regression-bs052.html | 331 ++++--- .../Regression/html/._Regression-bs053.html | 342 +++---- .../Regression/html/._Regression-bs054.html | 307 +++--- .../Regression/html/._Regression-bs055.html | 320 +++---- .../Regression/html/._Regression-bs056.html | 340 ++++--- .../Regression/html/._Regression-bs057.html | 332 ++++--- .../Regression/html/._Regression-bs058.html | 303 +++--- .../Regression/html/._Regression-bs059.html | 316 +++--- .../Regression/html/._Regression-bs060.html | 304 +++--- .../Regression/html/._Regression-bs061.html | 303 +++--- .../Regression/html/._Regression-bs062.html | 306 +++--- .../Regression/html/._Regression-bs063.html | 315 +++--- .../Regression/html/._Regression-bs064.html | 303 +++--- .../Regression/html/._Regression-bs065.html | 314 +++--- .../Regression/html/._Regression-bs066.html | 312 +++--- .../Regression/html/._Regression-bs067.html | 308 +++--- .../Regression/html/._Regression-bs068.html | 303 +++--- .../Regression/html/._Regression-bs069.html | 314 +++--- .../Regression/html/._Regression-bs070.html | 314 +++--- .../Regression/html/._Regression-bs071.html | 308 +++--- .../Regression/html/._Regression-bs072.html | 325 ++++--- .../Regression/html/._Regression-bs073.html | 314 +++--- .../Regression/html/._Regression-bs074.html | 327 +++---- .../Regression/html/._Regression-bs075.html | 373 ++++--- .../Regression/html/._Regression-bs076.html | 334 ++++--- .../Regression/html/._Regression-bs077.html | 298 +++--- .../Regression/html/._Regression-bs078.html | 324 +++---- .../Regression/html/._Regression-bs079.html | 375 ++++---- .../Regression/html/._Regression-bs080.html | 321 +++---- .../Regression/html/._Regression-bs081.html | 297 +++--- .../Regression/html/._Regression-bs082.html | 327 ++++--- .../Regression/html/._Regression-bs083.html | 328 +++---- .../Regression/html/._Regression-bs084.html | 305 +++--- .../Regression/html/._Regression-bs085.html | 305 +++--- .../Regression/html/._Regression-bs086.html | 323 +++---- .../Regression/html/._Regression-bs087.html | 309 +++--- .../Regression/html/._Regression-bs088.html | 346 ++++--- .../Regression/html/._Regression-bs089.html | 384 +++++--- .../Regression/html/._Regression-bs090.html | 349 ++++--- .../Regression/html/._Regression-bs091.html | 351 ++++--- .../Regression/html/._Regression-bs092.html | 368 ++++--- .../Regression/html/._Regression-bs093.html | 398 ++++---- .../Regression/html/._Regression-bs094.html | 391 ++++---- .../Regression/html/._Regression-bs095.html | 373 ++++--- .../Regression/html/._Regression-bs096.html | 368 ++++--- .../Regression/html/._Regression-bs097.html | 342 +++---- .../Regression/html/._Regression-bs098.html | 418 ++++---- .../Regression/html/._Regression-bs099.html | 278 +++--- .../Regression/html/._Regression-bs100.html | 278 +++--- .../Regression/html/._Regression-bs101.html | 278 +++--- .../Regression/html/._Regression-bs102.html | 278 +++--- .../Regression/html/._Regression-bs103.html | 278 +++--- doc/pub/Regression/html/Regression-bs.html | 280 +++--- .../Regression/html/Regression-reveal.html | 718 +++++++------- .../Regression/html/Regression-solarized.html | 786 ++++++++------- doc/pub/Regression/html/Regression.html | 786 ++++++++------- doc/pub/Regression/ipynb/Regression.ipynb | 906 +++++++++--------- .../ipynb/ipynb-Regression-src.tar.gz | Bin 211 -> 211 bytes doc/pub/Regression/pdf/Regression-minted.pdf | Bin 451746 -> 451753 bytes doc/src/Regression/Regression.do.txt | 432 ++++----- 112 files changed, 18180 insertions(+), 18192 deletions(-) diff --git a/doc/pub/Regression/html/._Regression-bs000.html b/doc/pub/Regression/html/._Regression-bs000.html index 786ff5b64..b5293106b 100644 --- a/doc/pub/Regression/html/._Regression-bs000.html +++ b/doc/pub/Regression/html/._Regression-bs000.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -402,7 +402,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Aug 29, 2019

    +

    Aug 30, 2019


    diff --git a/doc/pub/Regression/html/._Regression-bs001.html b/doc/pub/Regression/html/._Regression-bs001.html index d905d84aa..9708d32ef 100644 --- a/doc/pub/Regression/html/._Regression-bs001.html +++ b/doc/pub/Regression/html/._Regression-bs001.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs002.html b/doc/pub/Regression/html/._Regression-bs002.html index 720625ee3..8f02d82f0 100644 --- a/doc/pub/Regression/html/._Regression-bs002.html +++ b/doc/pub/Regression/html/._Regression-bs002.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs003.html b/doc/pub/Regression/html/._Regression-bs003.html index 761ee7817..9cac48f39 100644 --- a/doc/pub/Regression/html/._Regression-bs003.html +++ b/doc/pub/Regression/html/._Regression-bs003.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs004.html b/doc/pub/Regression/html/._Regression-bs004.html index a8191158c..c6d617952 100644 --- a/doc/pub/Regression/html/._Regression-bs004.html +++ b/doc/pub/Regression/html/._Regression-bs004.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs005.html b/doc/pub/Regression/html/._Regression-bs005.html index e77c3c692..7f18e57d8 100644 --- a/doc/pub/Regression/html/._Regression-bs005.html +++ b/doc/pub/Regression/html/._Regression-bs005.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs006.html b/doc/pub/Regression/html/._Regression-bs006.html index 90eb666d9..69a2cfe84 100644 --- a/doc/pub/Regression/html/._Regression-bs006.html +++ b/doc/pub/Regression/html/._Regression-bs006.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs007.html b/doc/pub/Regression/html/._Regression-bs007.html index 5634ba805..19f0763a0 100644 --- a/doc/pub/Regression/html/._Regression-bs007.html +++ b/doc/pub/Regression/html/._Regression-bs007.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs008.html b/doc/pub/Regression/html/._Regression-bs008.html index 2274ff3e2..5ab9ebacf 100644 --- a/doc/pub/Regression/html/._Regression-bs008.html +++ b/doc/pub/Regression/html/._Regression-bs008.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs009.html b/doc/pub/Regression/html/._Regression-bs009.html index fe8fa5429..d12c98bd8 100644 --- a/doc/pub/Regression/html/._Regression-bs009.html +++ b/doc/pub/Regression/html/._Regression-bs009.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs010.html b/doc/pub/Regression/html/._Regression-bs010.html index bf9ce1fed..3f8a9bb59 100644 --- a/doc/pub/Regression/html/._Regression-bs010.html +++ b/doc/pub/Regression/html/._Regression-bs010.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs011.html b/doc/pub/Regression/html/._Regression-bs011.html index 4fd62faf4..9dcc70156 100644 --- a/doc/pub/Regression/html/._Regression-bs011.html +++ b/doc/pub/Regression/html/._Regression-bs011.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs012.html b/doc/pub/Regression/html/._Regression-bs012.html index 775438c9b..5dde47771 100644 --- a/doc/pub/Regression/html/._Regression-bs012.html +++ b/doc/pub/Regression/html/._Regression-bs012.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs013.html b/doc/pub/Regression/html/._Regression-bs013.html index 45de12e02..e8d0a6a5b 100644 --- a/doc/pub/Regression/html/._Regression-bs013.html +++ b/doc/pub/Regression/html/._Regression-bs013.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs014.html b/doc/pub/Regression/html/._Regression-bs014.html index 15e5125f6..367ccd0e0 100644 --- a/doc/pub/Regression/html/._Regression-bs014.html +++ b/doc/pub/Regression/html/._Regression-bs014.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs015.html b/doc/pub/Regression/html/._Regression-bs015.html index a519c661a..5ac0d6272 100644 --- a/doc/pub/Regression/html/._Regression-bs015.html +++ b/doc/pub/Regression/html/._Regression-bs015.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs016.html b/doc/pub/Regression/html/._Regression-bs016.html index e533de6e2..ae0a00af2 100644 --- a/doc/pub/Regression/html/._Regression-bs016.html +++ b/doc/pub/Regression/html/._Regression-bs016.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs017.html b/doc/pub/Regression/html/._Regression-bs017.html index 91bd675bb..f2ae569c8 100644 --- a/doc/pub/Regression/html/._Regression-bs017.html +++ b/doc/pub/Regression/html/._Regression-bs017.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs018.html b/doc/pub/Regression/html/._Regression-bs018.html index b5ddefa4e..866b3bf5f 100644 --- a/doc/pub/Regression/html/._Regression-bs018.html +++ b/doc/pub/Regression/html/._Regression-bs018.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs019.html b/doc/pub/Regression/html/._Regression-bs019.html index cd94396dc..59099e694 100644 --- a/doc/pub/Regression/html/._Regression-bs019.html +++ b/doc/pub/Regression/html/._Regression-bs019.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs020.html b/doc/pub/Regression/html/._Regression-bs020.html index ced9e0bb6..af8d65e55 100644 --- a/doc/pub/Regression/html/._Regression-bs020.html +++ b/doc/pub/Regression/html/._Regression-bs020.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs021.html b/doc/pub/Regression/html/._Regression-bs021.html index 0b4be1765..46ebf3b80 100644 --- a/doc/pub/Regression/html/._Regression-bs021.html +++ b/doc/pub/Regression/html/._Regression-bs021.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs022.html b/doc/pub/Regression/html/._Regression-bs022.html index 33fd33d3a..0cc67fc3c 100644 --- a/doc/pub/Regression/html/._Regression-bs022.html +++ b/doc/pub/Regression/html/._Regression-bs022.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs023.html b/doc/pub/Regression/html/._Regression-bs023.html index f091d893a..ce7e28538 100644 --- a/doc/pub/Regression/html/._Regression-bs023.html +++ b/doc/pub/Regression/html/._Regression-bs023.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs024.html b/doc/pub/Regression/html/._Regression-bs024.html index 73311fad7..726f3a2b9 100644 --- a/doc/pub/Regression/html/._Regression-bs024.html +++ b/doc/pub/Regression/html/._Regression-bs024.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs025.html b/doc/pub/Regression/html/._Regression-bs025.html index 37f881698..5c4573a72 100644 --- a/doc/pub/Regression/html/._Regression-bs025.html +++ b/doc/pub/Regression/html/._Regression-bs025.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs026.html b/doc/pub/Regression/html/._Regression-bs026.html index 2c346d461..effe8cf1b 100644 --- a/doc/pub/Regression/html/._Regression-bs026.html +++ b/doc/pub/Regression/html/._Regression-bs026.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs027.html b/doc/pub/Regression/html/._Regression-bs027.html index 26c600e40..1b61aa69f 100644 --- a/doc/pub/Regression/html/._Regression-bs027.html +++ b/doc/pub/Regression/html/._Regression-bs027.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs028.html b/doc/pub/Regression/html/._Regression-bs028.html index b1a3a2b1a..e754349d2 100644 --- a/doc/pub/Regression/html/._Regression-bs028.html +++ b/doc/pub/Regression/html/._Regression-bs028.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs029.html b/doc/pub/Regression/html/._Regression-bs029.html index 938ec03ad..a470c974a 100644 --- a/doc/pub/Regression/html/._Regression-bs029.html +++ b/doc/pub/Regression/html/._Regression-bs029.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • diff --git a/doc/pub/Regression/html/._Regression-bs030.html b/doc/pub/Regression/html/._Regression-bs030.html index 51f31975b..0d4a702bc 100644 --- a/doc/pub/Regression/html/._Regression-bs030.html +++ b/doc/pub/Regression/html/._Regression-bs030.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -395,8 +395,7 @@ The examples we have looked at so far are cases where we normally can invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we did both for the masses and the fitting of the equation of state, leads to row vectors of the design matrix which are essentially -orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky decomposition. -More material to come here. +orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky decomposition.

    This may @@ -417,9 +416,6 @@ inversion algorithm. Thereafter we dive into the math of the SVD. -

    - -

    diff --git a/doc/pub/Regression/html/._Regression-bs031.html b/doc/pub/Regression/html/._Regression-bs031.html index a2bf95958..0c1e209c9 100644 --- a/doc/pub/Regression/html/._Regression-bs031.html +++ b/doc/pub/Regression/html/._Regression-bs031.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,59 +383,53 @@ MathJax.Hub.Config({ -

    The Ising model

    +

    Linear Regression Problems

    -The one-dimensional Ising model with nearest neighbor interaction, no -external field and a constant coupling constant \( J \) is given by - +One of the typical problems we encounter with linear regression, in particular +when the matrix \( \boldsymbol{X} \) (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of \( \boldsymbol{X} \) +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix $$ -\begin{align} - H = -J \sum_{k}^L s_k s_{k + 1}, -\tag{1} -\end{align} +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} $$

    -where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins -in the system is determined by \( L \). For the one-dimensional system -there is no phase transition. +The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2.

    -We will look at a system of \( L = 40 \) spins with a coupling constant of -\( J = 1 \). To get enough training data we will generate 10000 states -with their respective energies. +Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies +that the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +$$ +\begin{align*} +\boldsymbol{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +$$ -

    - - -

    import numpy as np
    -import matplotlib.pyplot as plt
    -from mpl_toolkits.axes_grid1 import make_axes_locatable
    -import seaborn as sns
    -import scipy.linalg as scl
    -from sklearn.model_selection import train_test_split
    -import tqdm
    -sns.set(color_codes=True)
    -cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
    -
    -L = 40
    -n = int(1e4)
    -
    -spins = np.random.choice([-1, 1], size=(n, L))
    -J = 1.0
    -
    -energies = np.zeros(n)
    -
    -for i in range(n):
    -    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
    -
    -

    -Here we use ordinary least squares -regression to predict the energy for the nearest neighbor -one-dimensional Ising model on a ring, i.e., the endpoints wrap -around. We will use linear regression to fit a value for -the coupling constant to achieve this. +We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. +This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least an eigenvalue which is zero.

    diff --git a/doc/pub/Regression/html/._Regression-bs032.html b/doc/pub/Regression/html/._Regression-bs032.html index 83c49de0e..3450418ad 100644 --- a/doc/pub/Regression/html/._Regression-bs032.html +++ b/doc/pub/Regression/html/._Regression-bs032.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,53 +383,31 @@ MathJax.Hub.Config({ -

    Reformulating the problem to suit regression

    +

    Fixing the singularity

    -A more general form for the one-dimensional Ising model is - +If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem $$ \begin{align} - H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. -\tag{2} +\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y}, +\tag{1} \end{align} $$ -

    -Here we allow for interactions beyond the nearest neighbors and a state dependent -coupling constant. This latter expression can be formulated as -a matrix-product -$$ -\begin{align} - \boldsymbol{H} = \boldsymbol{X} J, -\tag{3} -\end{align} -$$ +has linearly dependent column vectors, we will not be able to compute the inverse +of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \). +The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. +This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters \( \beta_i \) cannot be estimated.

    -where \( X_{jk} = s_j s_k \) and \( J \) is a matrix which consists of the -elements \( -J_{jk} \). This form of writing the energy fits perfectly -with the form utilized in linear regression, that is - +A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change $$ -\begin{align} - \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}, -\tag{4} -\end{align} +\boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I}, $$ -

    -We split the data in training and test data as discussed in the previous example +where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later. -

    - - -

    X = np.zeros((n, L ** 2))
    -for i in range(n):
    -    X[i] = np.outer(spins[i], spins[i]).ravel()
    -y = energies
    -X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
    -

    diff --git a/doc/pub/Regression/html/._Regression-bs033.html b/doc/pub/Regression/html/._Regression-bs033.html index 64b27e207..f9723f764 100644 --- a/doc/pub/Regression/html/._Regression-bs033.html +++ b/doc/pub/Regression/html/._Regression-bs033.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,51 +383,42 @@ MathJax.Hub.Config({ -

    Linear regression

    +

    Basic math of the SVD

    -In the ordinary least squares method we choose the cost function +From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) +we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). +The matrix has then a set of eigenpairs $$ -\begin{align} - C(\boldsymbol{X}, \boldsymbol{\beta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})\right\}. -\tag{5} -\end{align} +(\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), $$ -

    -We then find the extremal point of \( C \) by taking the derivative with respect to \( \boldsymbol{\beta} \) as discussed above. -This yields the expression for \( \boldsymbol{\beta} \) to be - +and the eigenvalues are given by the diagonal matrix $$ - \boldsymbol{\beta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}}, +\boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). $$ -

    -which immediately imposes some requirements on \( \boldsymbol{X} \) as there must exist -an inverse of \( \boldsymbol{X}^T \boldsymbol{X} \). If the expression we are modeling contains an -intercept, i.e., a constant term, we must make sure that the -first column of \( \boldsymbol{X} \) consists of \( 1 \). We do this here +The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \) +$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \).

    +Not all square matrices are diagonalizable. A matrix like the one discussed above +$$ +\boldsymbol{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +$$ - -

    X_train_own = np.concatenate(
    -    (np.ones(len(X_train))[:, np.newaxis], X_train),
    -    axis=1
    -)
    -X_test_own = np.concatenate(
    -    (np.ones(len(X_test))[:, np.newaxis], X_test),
    -    axis=1
    -)
    -
    -

    +is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition +\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. - -

    def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray:
    -    return scl.inv(x.T @ x) @ (x.T @ y)
    -beta = ols_inv(X_train_own, y_train)
    -

    diff --git a/doc/pub/Regression/html/._Regression-bs034.html b/doc/pub/Regression/html/._Regression-bs034.html index 45d40727a..6c36399ff 100644 --- a/doc/pub/Regression/html/._Regression-bs034.html +++ b/doc/pub/Regression/html/._Regression-bs034.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,89 +383,33 @@ MathJax.Hub.Config({ -

    Singular Value decomposition

    +

    The SVD, a Fantastic Algorithm

    -Doing the inversion directly turns out to be a bad idea since the matrix -\( \boldsymbol{X}^T\boldsymbol{X} \) is singular. An alternative approach is to use the singular -value decomposition. Using the definition of the Moore-Penrose -pseudoinverse we can write the equation for \( \boldsymbol{\beta} \) as +However, and this is the strength of the SVD algorithm, any general +matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The Singular Value Decompostion +(SVD) theorem +states that a general \( m\times n \) matrix \( \boldsymbol{X} \) can be written in +terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( n\times n \) +and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has +dimensionality \( m \times m \) and the last dimensionality \( n\times n \). +We have then -$$ - \boldsymbol{\beta} = \boldsymbol{X}^{+}\boldsymbol{y}, +$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T $$

    -where the pseudoinverse of \( \boldsymbol{X} \) is given by +As an example, the above defective matrix can be decomposed as $$ - \boldsymbol{X}^{+} = \frac{\boldsymbol{X}^T}{\boldsymbol{X}^T\boldsymbol{X}}. +\boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$

    -Using singular value decomposition we can decompose the matrix \( \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma} \boldsymbol{V}^T \), -where \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal(unitary) matrices and \( \boldsymbol{\Sigma} \) contains the singular values (more details below). -where \( X^{+} = V\Sigma^{+} U^T \). This reduces the equation for -\( \omega \) to -$$ -\begin{align} - \boldsymbol{\beta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}. -\tag{6} -\end{align} -$$ - -

    -Note that solving this equation by actually doing the pseudoinverse -(which is what we will do) is not a good idea as this operation scales -as \( \mathcal{O}(n^3) \), where \( n \) is the number of elements in a -general matrix. Instead, doing \( QR \)-factorization and solving the -linear system as an equation would reduce this down to -\( \mathcal{O}(n^2) \) operations. - -

    - - -

    def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray:
    -    u, s, v = scl.svd(x)
    -    return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y
    -
    -

    - - -

    beta = ols_svd(X_train_own,y_train)
    -
    -

    -When extracting the \( J \)-matrix we need to make sure that we remove the intercept, as is done here - -

    - - -

    J = beta[1:].reshape(L, L)
    -
    -

    -A way of looking at the coefficients in \( J \) is to plot the matrices as images. - -

    - - -

    fig = plt.figure(figsize=(20, 14))
    -im = plt.imshow(J, **cmap_args)
    -plt.title("OLS", fontsize=18)
    -plt.xticks(fontsize=18)
    -plt.yticks(fontsize=18)
    -cb = fig.colorbar(im)
    -cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
    -plt.show()
    -
    -

    -It is interesting to note that OLS -considers both \( J_{j, j + 1} = -0.5 \) and \( J_{j, j - 1} = -0.5 \) as -valid matrix elements for \( J \). -In our discussion below on hyperparameters and Ridge and Lasso regression we will see that -this problem can be removed, partly and only with Lasso regression. - -

    -In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD? +with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). +The SVD exits always!

    diff --git a/doc/pub/Regression/html/._Regression-bs035.html b/doc/pub/Regression/html/._Regression-bs035.html index 23a45041a..78c5abab6 100644 --- a/doc/pub/Regression/html/._Regression-bs035.html +++ b/doc/pub/Regression/html/._Regression-bs035.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,53 +383,40 @@ MathJax.Hub.Config({ -

    Linear Regression Problems

    +

    Another Example

    -One of the typical problems we encounter with linear regression, in particular -when the matrix \( \boldsymbol{X} \) (our so-called design matrix) is high-dimensional, -are problems with near singular or singular matrices. The column vectors of \( \boldsymbol{X} \) -may be linearly dependent, normally referred to as super-collinearity. -This means that the matrix may be rank deficient and it is basically impossible to -to model the data using linear regression. As an example, consider the matrix +Consider the following matrix which can be SVD decomposed as + $$ -\begin{align*} -\mathbf{X} & = \left[ -\begin{array}{rrr} -1 & -1 & 2 -\\ -1 & 0 & 1 -\\ -1 & 2 & -1 -\\ -1 & 1 & 0 -\end{array} \right] -\end{align*} +\boldsymbol{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{bmatrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. $$

    -The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the -the first column is the row-wise sum of the other two columns. The rank (more correct, -the column rank) of a matrix is the dimension of the space spanned by the -column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number -of linearly independent columns. In this particular case the matrix has rank 2. +This is a \( 3\times 2 \) matrix which is decomposed in terms of a +\( 3\times 3 \) matrix \( \boldsymbol{U} \), and a \( 2\times 2 \) matrix \( \boldsymbol{V} \). It is easy to see +that \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal (how?).

    -Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies -that the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{x} \) (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this -$$ -\begin{align*} -\boldsymbol{X} & = \left[ -\begin{array}{rr} -1 & -1 -\\ -1 & -1 -\end{array} \right]. -\end{align*} -$$ +And the SVD +decomposition (singular values) gives eigenvalues +\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=2 \), the +eigenvalues (singular values) are zero. -We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. -This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least an eigenvalue which is zero. +

    +In the general case, where our design matrix \( \boldsymbol{X} \) has dimension +\( n\times p \), the matrix is thus decomposed into an \( n\times n \) +orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) +and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) +singular values \( \sigma_i\geq 0 \) on the main diagonal and zeros filling +the rest of the matrix. There are at most \( p \) singular values +assuming that \( n > p \). In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have \( p > n \). These are often cases that lead to +near singular or singular matrices. + +

    +The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors.

    diff --git a/doc/pub/Regression/html/._Regression-bs036.html b/doc/pub/Regression/html/._Regression-bs036.html index df2e0707a..73ca49000 100644 --- a/doc/pub/Regression/html/._Regression-bs036.html +++ b/doc/pub/Regression/html/._Regression-bs036.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,30 +383,27 @@ MathJax.Hub.Config({ -

    Fixing the singularity

    +

    Economy-size SVD

    -If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem -$$ -\begin{align} -\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y}, -\tag{7} -\end{align} -$$ - -has linearly dependent column vectors, we will not be able to compute the inverse -of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \). -The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. -This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where -the regression parameters \( \beta_i \) cannot be estimated. +If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n +\times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however +irrelevant in our calculations since they are multiplied with the +zeros in \( \boldsymbol{\Sigma} \).

    -A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change -$$ -\boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I}, -$$ +The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns +in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition. -where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later. +

    +If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). +If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). +The \( n=p \) case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy.

    diff --git a/doc/pub/Regression/html/._Regression-bs037.html b/doc/pub/Regression/html/._Regression-bs037.html index 400c5aebf..6142e651b 100644 --- a/doc/pub/Regression/html/._Regression-bs037.html +++ b/doc/pub/Regression/html/._Regression-bs037.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,41 +383,54 @@ MathJax.Hub.Config({ -

    Basic math of the SVD

    +

    Mathematical Properties

    -From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is -a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) -we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). -The matrix has then a set of eigenpairs - -$$ -(\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), -$$ - -and the eigenvalues are given by the diagonal matrix -$$ -\boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). -$$ - -The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \) -$$ -\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, -$$ - -with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \). +There are several interesting mathematical properties which will be +relevant when we are going to discuss the differences between say +ordinary least squares (OLS) and Ridge regression.

    -Not all square matrices are diagonalizable. A matrix like the one discussed above +We have from OLS that the parameters of the linear approximation are given by $$ -\boldsymbol{X} = \begin{bmatrix} -1& -1 \\ -1& -1\\ -\end{bmatrix} +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition -\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. +

    +The matrix to invert can be rewritten in terms of our SVD decomposition as + +$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +Using the orthogonality properties of \( \boldsymbol{U} \) we have + +$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T, +$$ + +with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared. + +

    +This means that +$$ +(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D}, +$$ + +that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}, +$$ + +that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same. + +

    +Going back to our OLS equation we have +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. +$$ + +We will come back to this expression when we discuss Ridge regression.

    diff --git a/doc/pub/Regression/html/._Regression-bs038.html b/doc/pub/Regression/html/._Regression-bs038.html index 13c587454..5b604d134 100644 --- a/doc/pub/Regression/html/._Regression-bs038.html +++ b/doc/pub/Regression/html/._Regression-bs038.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,33 +383,60 @@ MathJax.Hub.Config({ -

    The SVD, a Fantastic Algorithm

    +

    Ridge and LASSO Regression

    -However, and this is the strength of the SVD algorithm, any general -matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and -two orthogonal/unitary matrices. The Singular Value Decompostion -(SVD) theorem -states that a general \( m\times n \) matrix \( \boldsymbol{X} \) can be written in -terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( n\times n \) -and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has -dimensionality \( m \times m \) and the last dimensionality \( n\times n \). -We have then +Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ -$$ -\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T +or we can state it as +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +$$ + +where we have used the definition of a norm-2 vector, that is +$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. $$

    -As an example, the above defective matrix can be decomposed as +By minimizing the above equation with respect to the parameters +\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is $$ -\boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 $$

    -with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). -The SVD exits always! +which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +$$ + +

    +we have a new optimization equation +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +

    +Here we have defined the norm-1 as +$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$

    diff --git a/doc/pub/Regression/html/._Regression-bs039.html b/doc/pub/Regression/html/._Regression-bs039.html index d01066aec..c63c5f3ef 100644 --- a/doc/pub/Regression/html/._Regression-bs039.html +++ b/doc/pub/Regression/html/._Regression-bs039.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,40 +383,61 @@ MathJax.Hub.Config({ -

    Another Example

    +

    More on Ridge Regression

    -Consider the following matrix which can be SVD decomposed as +Using the matrix-vector expression for Ridge regression, $$ -\boldsymbol{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{bmatrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$

    -This is a \( 3\times 2 \) matrix which is decomposed in terms of a -\( 3\times 3 \) matrix \( \boldsymbol{U} \), and a \( 2\times 2 \) matrix \( \boldsymbol{V} \). It is easy to see -that \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal (how?). +by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$

    -And the SVD -decomposition (singular values) gives eigenvalues -\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=2 \), the -eigenvalues (singular values) are zero. +with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that + +$$ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +$$

    -In the general case, where our design matrix \( \boldsymbol{X} \) has dimension -\( n\times p \), the matrix is thus decomposed into an \( n\times n \) -orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) -and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) -singular values \( \sigma_i\lg 0 \) on the main diagonal and zeros filling -the rest of the matrix. There are at most \( p \) singular values -assuming that \( n > p \). In our regression examples for the nuclear -masses and the equation of state this is indeed the case, while for -the Ising model we have \( p > n \). These are often cases that lead to -near singular or singular matrices. +with \( t \) a finite positive number.

    -The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors. +We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The +consequences, in particular for our discussion of the bias-variance +are rather interesting. + +

    +Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +$$ + +

    +We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +$$ + +

    +For Ridge regression this becomes + +$$ +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +$$ + +

    +with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \).

    diff --git a/doc/pub/Regression/html/._Regression-bs040.html b/doc/pub/Regression/html/._Regression-bs040.html index d1ac113a7..7fd678478 100644 --- a/doc/pub/Regression/html/._Regression-bs040.html +++ b/doc/pub/Regression/html/._Regression-bs040.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,27 +383,26 @@ MathJax.Hub.Config({ -

    Economy-size SVD

    +

    Interpreting the Ridge results

    -If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n -\times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however -irrelevant in our calculations since they are multiplied with the -zeros in \( \boldsymbol{\Sigma} \). +Since \( \lambda \geq 0 \), it means that compared to OLS, we have + +$$ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +$$

    -The economy-size decomposition removes extra rows or columns of zeros -from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns -in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. -Removing these zeros and columns can improve execution time -and reduce storage requirements without compromising the accuracy of -the decomposition. +Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \).

    -If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). -If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). -The \( n=p \) case is obvious, we retain the full SVD. -In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. +For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). +With a parameter \( \lambda \) we can thus shrink the role of specific parameters.

    diff --git a/doc/pub/Regression/html/._Regression-bs041.html b/doc/pub/Regression/html/._Regression-bs041.html index 2f58af5f0..ab77a840d 100644 --- a/doc/pub/Regression/html/._Regression-bs041.html +++ b/doc/pub/Regression/html/._Regression-bs041.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,54 +383,39 @@ MathJax.Hub.Config({ -

    Mathematical Properties

    +

    More interpretations

    -There are several interesting mathematical properties which will be -relevant when we are going to discuss the differences between say -ordinary least squares (OLS) and Ridge regression. +For the sake of simplicity, let us assume that the design matrix is orthonormal, that is -

    -We have from OLS that the parameters of the linear approximation are given by $$ -\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. $$

    -The matrix to invert can be rewritten in terms of our SVD decomposition as - +In this case the standard OLS results in $$ -\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, $$ -Using the orthogonality properties of \( \boldsymbol{U} \) we have - -$$ -\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T, -$$ - -with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared. -

    -This means that -$$ -(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D}, -$$ +and -that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that $$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}, +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, $$ -that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same. -

    -Going back to our OLS equation we have -$$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. -$$ +that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. -We will come back to this expression when we discuss Ridge regression. +

    +We will come back to more interpreations after we have gone through some of the statistical analysis part. + +

    +For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended.

    diff --git a/doc/pub/Regression/html/._Regression-bs042.html b/doc/pub/Regression/html/._Regression-bs042.html index 49f2db109..8d2b5faee 100644 --- a/doc/pub/Regression/html/._Regression-bs042.html +++ b/doc/pub/Regression/html/._Regression-bs042.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,60 +383,19 @@ MathJax.Hub.Config({ -

    Ridge and LASSO Regression

    +

    Where are we going?

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is -our optimization problem is -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. -$$ +Before we proceed, we need to rethink what we have been doing. In our +eager to fit the data, we have omitted several important elements in +our regression analysis. In what follows we will -or we can state it as -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, -$$ +

      +
    1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
    2. +
    3. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
    4. +
    -where we have used the definition of a norm-2 vector, that is -$$ -\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. -$$ - -

    -By minimizing the above equation with respect to the parameters -\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the -parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by -defining a new cost function to be optimized, that is - -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 -$$ - -

    -which leads to the Ridge regression minimization problem where we -require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is -a finite number larger than zero. By defining - -$$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, -$$ - -

    -we have a new optimization equation -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 -$$ - -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. - -

    -Here we have defined the norm-1 as -$$ -\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. -$$ +This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.

    diff --git a/doc/pub/Regression/html/._Regression-bs043.html b/doc/pub/Regression/html/._Regression-bs043.html index 857579026..416da079f 100644 --- a/doc/pub/Regression/html/._Regression-bs043.html +++ b/doc/pub/Regression/html/._Regression-bs043.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,61 +383,23 @@ MathJax.Hub.Config({ -

    More on Ridge Regression

    +

    Resampling methods

    +
    +
    +

    +Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +

    +
    -

    -Using the matrix-vector expression for Ridge regression, - -$$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, -$$ - -

    -by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then -a slightly modified matrix inversion problem which for finite values -of \( \lambda \) does not suffer from singularity problems. We obtain - -$$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ - -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that - -$$ -\sum_{i=0}^{p-1} \beta_i^2 \leq t, -$$ - -

    -with \( t \) a finite positive number. - -

    -We see that Ridge regression is nothing but the standard -OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The -consequences, in particular for our discussion of the bias-variance -are rather interesting. - -

    -Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had -$$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. -$$ - -

    -We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as -$$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} -$$ - -

    -For Ridge regression this becomes - -$$ -\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, -$$ - -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \).

    diff --git a/doc/pub/Regression/html/._Regression-bs044.html b/doc/pub/Regression/html/._Regression-bs044.html index 564661628..d3532c4de 100644 --- a/doc/pub/Regression/html/._Regression-bs044.html +++ b/doc/pub/Regression/html/._Regression-bs044.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,26 +383,32 @@ MathJax.Hub.Config({ -

    Interpreting the Ridge results

    +

    Resampling approaches can be computationally expensive

    +
    +
    +

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have - -$$ -\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. -$$ +Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used.

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the -orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by -\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has -eigenvalues ordered in a descending way, that is \( \sigma_i \geq -\sigma_{i+1} \). +

    +
    -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. -Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). -With a parameter \( \lambda \) we can thus shrink the role of specific parameters.

    diff --git a/doc/pub/Regression/html/._Regression-bs045.html b/doc/pub/Regression/html/._Regression-bs045.html index c8b667d8b..a2680f2fe 100644 --- a/doc/pub/Regression/html/._Regression-bs045.html +++ b/doc/pub/Regression/html/._Regression-bs045.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,39 +383,19 @@ MathJax.Hub.Config({ -

    More interpretations

    +

    Why resampling methods ?

    +
    +
    +

    -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is +

      +
    • Our simulations can be treated as computer experiments. This is particularly the case for Monte Carlo methods
    • +
    • The results can be analysed with the same statistical tools as we would use analysing experimental data.
    • +
    • As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
    • +
    +
    +
    -$$ -\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. -$$ - -

    -In this case the standard OLS results in -$$ -\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, -$$ - -

    -and - -$$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, -$$ - -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and -the Ridge estimator converges to zero when the hyperparameter goes to -infinity. - -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. - -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. -Similarly, Mehta et al's article is also recommended.

    diff --git a/doc/pub/Regression/html/._Regression-bs046.html b/doc/pub/Regression/html/._Regression-bs046.html index 4ae502f5b..05142a991 100644 --- a/doc/pub/Regression/html/._Regression-bs046.html +++ b/doc/pub/Regression/html/._Regression-bs046.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,19 +383,25 @@ MathJax.Hub.Config({ -

    Where are we going?

    +

    Statistical analysis

    +
    +
    +

    -

    -Before we proceed, we need to rethink what we have been doing. In our -eager to fit the data, we have omitted several important elements in -our regression analysis. In what follows we will +

      +
    • As in other experiments, many numerical experiments have two classes of errors:
    • -
        -
      1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
      2. -
      3. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
      4. -
      +
        +
      • Statistical errors
      • +
      • Systematical errors
      • +
      + +
    • Statistical errors can be estimated using standard tools from statistics
    • +
    • Systematical errors are method specific and must be treated differently from case to case.
    • +
    +
    +
    -This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.

    diff --git a/doc/pub/Regression/html/._Regression-bs047.html b/doc/pub/Regression/html/._Regression-bs047.html index a8564ed23..7fa3a3ae5 100644 --- a/doc/pub/Regression/html/._Regression-bs047.html +++ b/doc/pub/Regression/html/._Regression-bs047.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,20 +383,31 @@ MathJax.Hub.Config({ -

    Resampling methods

    +

    Statistics

    -Resampling methods are an indispensable tool in modern -statistics. They involve repeatedly drawing samples from a training -set and refitting a model of interest on each sample in order to -obtain additional information about the fitted model. For example, in -order to estimate the variability of a linear regression fit, we can -repeatedly draw different samples from the training data, fit a linear -regression to each new sample, and then examine the extent to which -the resulting fits differ. Such an approach may allow us to obtain -information that would not be available from fitting the model only -once using the original training sample. +The probability distribution function (PDF) is a function +\( p(x) \) on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of \( X \) occur: +$$ +p(x) = \mathrm{prob}(X=x) +$$ + +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around \( x \) to be \( p(x)dx \). The continuous function \( p(x) \) then gives us +the density of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval \( [a,\,b] \) is then just the integral: +$$ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +$$ + +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF.

    diff --git a/doc/pub/Regression/html/._Regression-bs048.html b/doc/pub/Regression/html/._Regression-bs048.html index 0ed4f8137..9d1aeebbf 100644 --- a/doc/pub/Regression/html/._Regression-bs048.html +++ b/doc/pub/Regression/html/._Regression-bs048.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,29 +383,23 @@ MathJax.Hub.Config({ -

    Resampling approaches can be computationally expensive

    +

    Statistics, moments

    +A particularly useful class of special expectation values are the +moments. The \( n \)-th moment of the PDF \( p \) is defined as +follows: +$$ +\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +$$ -

    -Resampling approaches can be computationally expensive, because they -involve fitting the same statistical method multiple times using -different subsets of the training data. However, due to recent -advances in computing power, the computational requirements of -resampling methods generally are not prohibitive. In this chapter, we -discuss two of the most commonly used resampling methods, -cross-validation and the bootstrap. Both methods are important tools -in the practical application of many statistical learning -procedures. For example, cross-validation can be used to estimate the -test error associated with a given statistical learning method in -order to evaluate its performance, or to select the appropriate level -of flexibility. The process of evaluating a model’s performance is -known as model assessment, whereas the process of selecting the proper -level of flexibility for a model is known as model selection. The -bootstrap is widely used. - -

    +The zero-th moment \( \langle 1\rangle \) is just the normalization condition of +\( p \). The first moment, \( \langle x\rangle \), is called the mean of \( p \) +and often denoted by the letter \( \mu \): +$$ +\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +$$

    diff --git a/doc/pub/Regression/html/._Regression-bs049.html b/doc/pub/Regression/html/._Regression-bs049.html index 98dd9b01a..b24a901cb 100644 --- a/doc/pub/Regression/html/._Regression-bs049.html +++ b/doc/pub/Regression/html/._Regression-bs049.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,16 +383,38 @@ MathJax.Hub.Config({ -

    Why resampling methods ?

    +

    Statistics, central moments

    +A special version of the moments is the set of central moments, +the n-th central moment defined as: +$$ +\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +$$ -

      -
    • Our simulations can be treated as computer experiments. This is particularly the case for Monte Carlo methods
    • -
    • The results can be analysed with the same statistical tools as we would use analysing experimental data.
    • -
    • As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
    • -
    +The zero-th and first central moments are both trivial, equal \( 1 \) and +\( 0 \), respectively. But the second central moment, known as the +variance of \( p \), is of particular interest. For the stochastic +variable \( X \), the variance is denoted as \( \sigma^2_X \) or \( \mathrm{var}(X) \): +$$ +\begin{align} +\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = +\int\! (x-\langle x\rangle)^2 p(x)\,dx +\tag{2}\\ +& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + + \langle x\rangle^2\right)p(x)\,dx +\tag{3}\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2 +\tag{4}\\ +& = \langle x^2\rangle - \langle x\rangle^2 +\tag{5} +\end{align} +$$ + +The square root of the variance, \( \sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle} \) is called the standard deviation of \( p \). It is clearly just the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the spread of \( p \) around its mean.
    diff --git a/doc/pub/Regression/html/._Regression-bs050.html b/doc/pub/Regression/html/._Regression-bs050.html index 1b93786c3..74018c6ae 100644 --- a/doc/pub/Regression/html/._Regression-bs050.html +++ b/doc/pub/Regression/html/._Regression-bs050.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,22 +383,31 @@ MathJax.Hub.Config({ -

    Statistical analysis

    +

    Statistics, covariance

    +Another important quantity is the so called covariance, a variant of +the above defined variance. Consider again the set \( \{X_i\} \) of \( n \) +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF \( P(x_1,\dots,x_n) \). The covariance of two +of the stochastic variables, \( X_i \) and \( X_j \), is defined as follows: +$$ +\begin{align} +\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\nonumber\\ +&= +\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, +P(x_1,\dots,x_n)\,dx_1\dots dx_n +\tag{6} +\end{align} +$$ -

      -
    • As in other experiments, many numerical experiments have two classes of errors:
    • - -
        -
      • Statistical errors
      • -
      • Systematical errors
      • -
      - -
    • Statistical errors can be estimated using standard tools from statistics
    • -
    • Systematical errors are method specific and must be treated differently from case to case.
    • -
    +with +$$ +\langle x_i\rangle = +\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +$$
    diff --git a/doc/pub/Regression/html/._Regression-bs051.html b/doc/pub/Regression/html/._Regression-bs051.html index 9edf6eddb..8a4c80b93 100644 --- a/doc/pub/Regression/html/._Regression-bs051.html +++ b/doc/pub/Regression/html/._Regression-bs051.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,31 +383,32 @@ MathJax.Hub.Config({ -

    Statistics

    +

    Statistics, more covariance

    -The probability distribution function (PDF) is a function -\( p(x) \) on the domain which, in the discrete case, gives us the -probability or relative frequency with which these values of \( X \) occur: +If we consider the above covariance as a matrix \( C_{ij}=\mathrm{cov}(X_i,\,X_j) \), then the diagonal elements are just the familiar +variances, \( C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i) \). It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. This is easy to show, keeping in mind the linearity of +the expectation value. Consider the stochastic variables \( X_i \) and +\( X_j \), (\( i\neq j \)): $$ -p(x) = \mathrm{prob}(X=x) +\begin{align} +\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\tag{7}\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle +\tag{8}\\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle +\tag{9}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle +\tag{10}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\tag{11} +\end{align} $$ - -In the continuous case, the PDF does not directly depict the -actual probability. Instead we define the probability for the -stochastic variable to assume any value on an infinitesimal interval -around \( x \) to be \( p(x)dx \). The continuous function \( p(x) \) then gives us -the density of the probability rather than the probability -itself. The probability for a stochastic variable to assume any value -on a non-infinitesimal interval \( [a,\,b] \) is then just the integral: -$$ -\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx -$$ - -Qualitatively speaking, a stochastic variable represents the values of -numbers chosen as if by chance from some specified PDF so that the -selection of a large set of these numbers reproduces this PDF.

    diff --git a/doc/pub/Regression/html/._Regression-bs052.html b/doc/pub/Regression/html/._Regression-bs052.html index 78ef192c7..715ea3015 100644 --- a/doc/pub/Regression/html/._Regression-bs052.html +++ b/doc/pub/Regression/html/._Regression-bs052.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,26 +383,51 @@ MathJax.Hub.Config({ -

    Statistics, moments

    -
    -
    -

    -A particularly useful class of special expectation values are the -moments. The \( n \)-th moment of the PDF \( p \) is defined as -follows: +

    Covariance example

    + +

    +Suppose we have defined three vectors \( \hat{x}, \hat{y}, \hat{z} \) with +\( n \) elements each. The covariance matrix is defined as + $$ -\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +\hat{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\ + \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\ + \sigma_{zx} & \sigma_{zy} & \sigma_{zz} + \end{bmatrix}, $$ -The zero-th moment \( \langle 1\rangle \) is just the normalization condition of -\( p \). The first moment, \( \langle x\rangle \), is called the mean of \( p \) -and often denoted by the letter \( \mu \): +where for example $$ -\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). $$ -

    -
    +

    +The Numpy function np.cov calculates the covariance elements using +the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have +the exact mean valu\ es. + +

    +The following simple function uses the np.vstack function which +takes each vector of dimension \( 1\times n \) and produces a \( 3\times n \) +matrix \( \hat{W} \) + +$$ +\hat{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\ + x_1 & y_1 & z_1 \\ + x_2 & y_2 & z_2 \\ + \dots & \dots & \dots \\ + x_{n-2} & y_{n-2} & z_{n-2} \\ + x_{n-1} & y_{n-1} & z_{n-1} + \end{bmatrix}, +$$ + +

    +which in turn is converted into into the \( 3\times 3 \) covariance matrix +\( \hat{\Sigma} \) via the Numpy function np.cov(). We note that we can +also calculate the mean value of each set of samples \( \hat{x} \) etc +using the Numpy function np.mean(x). We can also extract the +eigenvalues of the covariance matrix through the np.linalg.eig() +function.

    diff --git a/doc/pub/Regression/html/._Regression-bs053.html b/doc/pub/Regression/html/._Regression-bs053.html index 597c51e87..108f03bca 100644 --- a/doc/pub/Regression/html/._Regression-bs053.html +++ b/doc/pub/Regression/html/._Regression-bs053.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,42 +383,42 @@ MathJax.Hub.Config({ -

    Statistics, central moments

    -
    -
    -

    -A special version of the moments is the set of central moments, -the n-th central moment defined as: -$$ -\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx -$$ +

    Covariance in numpy

    -The zero-th and first central moments are both trivial, equal \( 1 \) and -\( 0 \), respectively. But the second central moment, known as the -variance of \( p \), is of particular interest. For the stochastic -variable \( X \), the variance is denoted as \( \sigma^2_X \) or \( \mathrm{var}(X) \): -$$ -\begin{align} -\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = -\int\! (x-\langle x\rangle)^2 p(x)\,dx -\tag{8}\\ -& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + - \langle x\rangle^2\right)p(x)\,dx -\tag{9}\\ -& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2 -\tag{10}\\ -& = \langle x^2\rangle - \langle x\rangle^2 -\tag{11} -\end{align} -$$ +

    -The square root of the variance, \( \sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle} \) is called the standard deviation of \( p \). It is clearly just the RMS (root-mean-square) -value of the deviation of the PDF from its mean value, interpreted -qualitatively as the spread of \( p \) around its mean. -

    -
    + +
    # Importing various packages
    +import numpy as np
     
    +n = 100
    +x = np.random.normal(size=n)
    +print(np.mean(x))
    +y = 4+3*x+np.random.normal(size=n)
    +print(np.mean(y))
    +z = x**3+np.random.normal(size=n)
    +print(np.mean(z))
    +W = np.vstack((x, y, z))
    +Sigma = np.cov(W)
    +print(Sigma)
    +Eigvals, Eigvecs = np.linalg.eig(Sigma)
    +print(Eigvals)
    +
    +

    + +

    import numpy as np
    +import matplotlib.pyplot as plt
    +from scipy import sparse
    +eye = np.eye(4)
    +print(eye)
    +sparse_mtx = sparse.csr_matrix(eye)
    +print(sparse_mtx)
    +x = np.linspace(-10,10,100)
    +y = np.sin(x)
    +plt.plot(x,y,marker='x')
    +plt.show()
    +

    diff --git a/doc/pub/Regression/html/._Regression-bs054.html b/doc/pub/Regression/html/._Regression-bs054.html index 2ca729a79..deb8b2308 100644 --- a/doc/pub/Regression/html/._Regression-bs054.html +++ b/doc/pub/Regression/html/._Regression-bs054.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,30 +383,25 @@ MathJax.Hub.Config({ -

    Statistics, covariance

    +

    Statistics, independent variables

    -Another important quantity is the so called covariance, a variant of -the above defined variance. Consider again the set \( \{X_i\} \) of \( n \) -stochastic variables (not necessarily uncorrelated) with the -multivariate PDF \( P(x_1,\dots,x_n) \). The covariance of two -of the stochastic variables, \( X_i \) and \( X_j \), is defined as follows: +If \( X_i \) and \( X_j \) are independent, we get +\( \langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle \), resulting in \( \mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j) \). + +

    +Also useful for us is the covariance of linear combinations of +stochastic variables. Let \( \{X_i\} \) and \( \{Y_i\} \) be two sets of +stochastic variables. Let also \( \{a_i\} \) and \( \{b_i\} \) be two sets of +scalars. Consider the linear combination: $$ -\begin{align} -\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle -\nonumber\\ -&= -\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, -P(x_1,\dots,x_n)\,dx_1\dots dx_n -\tag{12} -\end{align} +U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j $$ -with +By the linearity of the expectation value $$ -\langle x_i\rangle = -\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) $$

    diff --git a/doc/pub/Regression/html/._Regression-bs055.html b/doc/pub/Regression/html/._Regression-bs055.html index d011d3b3b..09eddaf4f 100644 --- a/doc/pub/Regression/html/._Regression-bs055.html +++ b/doc/pub/Regression/html/._Regression-bs055.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,32 +383,32 @@ MathJax.Hub.Config({ -

    Statistics, more covariance

    +

    Statistics, more variance

    -If we consider the above covariance as a matrix \( C_{ij}=\mathrm{cov}(X_i,\,X_j) \), then the diagonal elements are just the familiar -variances, \( C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i) \). It turns out that -all the off-diagonal elements are zero if the stochastic variables are -uncorrelated. This is easy to show, keeping in mind the linearity of -the expectation value. Consider the stochastic variables \( X_i \) and -\( X_j \), (\( i\neq j \)): +Now, since the variance is just \( \mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i) \), we get +the variance of the linear combination \( U = \sum_i a_i X_i \): $$ -\begin{align} -\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle -\tag{13}\\ -&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle -\tag{14}\\ -&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + -\langle \langle x_i\rangle\langle x_j\rangle\rangle -\tag{15}\\ -&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + -\langle x_i\rangle\langle x_j\rangle -\tag{16}\\ -&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle -\tag{17} -\end{align} +\begin{equation} +\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) +\tag{12} +\end{equation} $$ + +And in the special case when the stochastic variables are +uncorrelated, the off-diagonal elements of the covariance are as we +know zero, resulting in: +$$ +\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ + +$$ +\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ + +which will become very useful in our study of the error in the mean +value of a set of measurements.

    diff --git a/doc/pub/Regression/html/._Regression-bs056.html b/doc/pub/Regression/html/._Regression-bs056.html index acbd35880..5ece4bfd0 100644 --- a/doc/pub/Regression/html/._Regression-bs056.html +++ b/doc/pub/Regression/html/._Regression-bs056.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,51 +383,31 @@ MathJax.Hub.Config({ -

    Covariance example

    - -

    -Suppose we have defined three vectors \( \hat{x}, \hat{y}, \hat{z} \) with -\( n \) elements each. The covariance matrix is defined as - +

    Statistics and stochastic processes

    +
    +
    +

    +A stochastic process is a process that produces sequentially a +chain of values: $$ -\hat{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\ - \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\ - \sigma_{zx} & \sigma_{zy} & \sigma_{zz} - \end{bmatrix}, +\{x_1, x_2,\dots\,x_k,\dots\}. $$ -where for example -$$ -\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). -$$ +We will call these +values our measurements and the entire set as our measured +sample. The action of measuring all the elements of a sample +we will call a stochastic experiment since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment. We +assume that these values are distributed according to some +PDF \( p_X^{\phantom X}(x) \), where \( X \) is just the formal symbol for the +stochastic variable whose PDF is \( p_X^{\phantom X}(x) \). Instead of +trying to determine the full distribution \( p \) we are often only +interested in finding the few lowest moments, like the mean +\( \mu_X^{\phantom X} \) and the variance \( \sigma_X^{\phantom X} \). +

    +
    -

    -The Numpy function np.cov calculates the covariance elements using -the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have -the exact mean valu\ es. - -

    -The following simple function uses the np.vstack function which -takes each vector of dimension \( 1\times n \) and produces a \( 3\times n \) -matrix \( \hat{W} \) - -$$ -\hat{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\ - x_1 & y_1 & z_1 \\ - x_2 & y_2 & z_2 \\ - \dots & \dots & \dots \\ - x_{n-2} & y_{n-2} & z_{n-2} \\ - x_{n-1} & y_{n-1} & z_{n-1} - \end{bmatrix}, -$$ - -

    -which in turn is converted into into the \( 3\times 3 \) covariance matrix -\( \hat{\Sigma} \) via the Numpy function np.cov(). We note that we can -also calculate the mean value of each set of samples \( \hat{x} \) etc -using the Numpy function np.mean(x). We can also extract the -eigenvalues of the covariance matrix through the np.linalg.eig() -function.

    diff --git a/doc/pub/Regression/html/._Regression-bs057.html b/doc/pub/Regression/html/._Regression-bs057.html index e969faf61..01bd4d7b1 100644 --- a/doc/pub/Regression/html/._Regression-bs057.html +++ b/doc/pub/Regression/html/._Regression-bs057.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -381,44 +381,32 @@ MathJax.Hub.Config({

     

     

     

    - + -

    Covariance in numpy

    +

    Statistics and sample variables

    +
    +
    +

    +In practical situations a sample is always of finite size. Let that +size be \( n \). The expectation value of a sample, the sample mean, is then defined as follows: +$$ +\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +$$ -

    +The sample variance is: +$$ +\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 +$$ - -

    # Importing various packages
    -import numpy as np
    +its square root being the standard deviation of the sample. The
    +sample covariance is:
    +$$
    +\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n)
    +$$
    +
    +
    -n = 100 -x = np.random.normal(size=n) -print(np.mean(x)) -y = 4+3*x+np.random.normal(size=n) -print(np.mean(y)) -z = x**3+np.random.normal(size=n) -print(np.mean(z)) -W = np.vstack((x, y, z)) -Sigma = np.cov(W) -print(Sigma) -Eigvals, Eigvecs = np.linalg.eig(Sigma) -print(Eigvals) -
    -

    - -

    import numpy as np
    -import matplotlib.pyplot as plt
    -from scipy import sparse
    -eye = np.eye(4)
    -print(eye)
    -sparse_mtx = sparse.csr_matrix(eye)
    -print(sparse_mtx)
    -x = np.linspace(-10,10,100)
    -y = np.sin(x)
    -plt.plot(x,y,marker='x')
    -plt.show()
    -

    diff --git a/doc/pub/Regression/html/._Regression-bs058.html b/doc/pub/Regression/html/._Regression-bs058.html index b4a0d26a7..280ac6b3d 100644 --- a/doc/pub/Regression/html/._Regression-bs058.html +++ b/doc/pub/Regression/html/._Regression-bs058.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,26 +383,21 @@ MathJax.Hub.Config({ -

    Statistics, independent variables

    +

    Statistics, sample variance and covariance

    -If \( X_i \) and \( X_j \) are independent, we get -\( \langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle \), resulting in \( \mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j) \). +Note that the sample variance is the sample covariance without the +cross terms. In a similar manner as the covariance in Eq. (6) is a measure of the correlation between +two stochastic variables, the above defined sample covariance is a +measure of the sequential correlation between succeeding measurements +of a sample.

    -Also useful for us is the covariance of linear combinations of -stochastic variables. Let \( \{X_i\} \) and \( \{Y_i\} \) be two sets of -stochastic variables. Let also \( \{a_i\} \) and \( \{b_i\} \) be two sets of -scalars. Consider the linear combination: -$$ -U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j -$$ - -By the linearity of the expectation value -$$ -\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) -$$ +These quantities, being known experimental values, differ +significantly from and must not be confused with the similarly named +quantities for stochastic variables, mean \( \mu_X \), variance \( \mathrm{var}(X) \) +and covariance \( \mathrm{cov}(X,Y) \).

    diff --git a/doc/pub/Regression/html/._Regression-bs059.html b/doc/pub/Regression/html/._Regression-bs059.html index 8382b0968..bd53f2d50 100644 --- a/doc/pub/Regression/html/._Regression-bs059.html +++ b/doc/pub/Regression/html/._Regression-bs059.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,32 +383,32 @@ MathJax.Hub.Config({ -

    Statistics, more variance

    +

    Statistics, law of large numbers

    -Now, since the variance is just \( \mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i) \), we get -the variance of the linear combination \( U = \sum_i a_i X_i \): +The law of large numbers +states that as the size of our sample grows to infinity, the sample +mean approaches the true mean \( \mu_X^{\phantom X} \) of the chosen PDF: $$ -\begin{equation} -\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) -\tag{18} -\end{equation} +\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} $$ -And in the special case when the stochastic variables are -uncorrelated, the off-diagonal elements of the covariance are as we -know zero, resulting in: -$$ -\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) -$$ +The sample mean \( \bar{x}_n \) works therefore as an estimate of the true +mean \( \mu_X^{\phantom X} \). -$$ -\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) -$$ - -which will become very useful in our study of the error in the mean -value of a set of measurements. +

    +What we need to find out is how good an approximation \( \bar{x}_n \) is to +\( \mu_X^{\phantom X} \). In any stochastic measurement, an estimated +mean is of no use to us without a measure of its error. A quantity +that tells us how well we can reproduce it in another experiment. We +are therefore interested in the PDF of the sample mean itself. Its +standard deviation will be a measure of the spread of sample means, +and we will simply call it the error of the sample mean, or +just sample error, and denote it by \( \mathrm{err}_X^{\phantom X} \). In +practice, we will only be able to produce an estimate of the +sample error since the exact value would require the knowledge of the +true PDFs behind, which we usually do not have.

    diff --git a/doc/pub/Regression/html/._Regression-bs060.html b/doc/pub/Regression/html/._Regression-bs060.html index e91dc22f6..db17155ca 100644 --- a/doc/pub/Regression/html/._Regression-bs060.html +++ b/doc/pub/Regression/html/._Regression-bs060.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,28 +383,22 @@ MathJax.Hub.Config({ -

    Statistics and stochastic processes

    +

    Statistics, more on sample error

    -A stochastic process is a process that produces sequentially a -chain of values: +Let us first take a look at what happens to the sample error as the +size of the sample grows. In a sample, each of the measurements \( x_i \) +can be associated with its own stochastic variable \( X_i \). The +stochastic variable \( \overline X_n \) for the sample mean \( \bar{x}_n \) is +then just a linear combination, already familiar to us: $$ -\{x_1, x_2,\dots\,x_k,\dots\}. +\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i $$ -We will call these -values our measurements and the entire set as our measured -sample. The action of measuring all the elements of a sample -we will call a stochastic experiment since, operationally, -they are often associated with results of empirical observation of -some physical or mathematical phenomena; precisely an experiment. We -assume that these values are distributed according to some -PDF \( p_X^{\phantom X}(x) \), where \( X \) is just the formal symbol for the -stochastic variable whose PDF is \( p_X^{\phantom X}(x) \). Instead of -trying to determine the full distribution \( p \) we are often only -interested in finding the few lowest moments, like the mean -\( \mu_X^{\phantom X} \) and the variance \( \sigma_X^{\phantom X} \). +All the coefficients are just equal \( 1/n \). The PDF of \( \overline X_n \), +denoted by \( p_{\overline X_n}(x) \) is the desired PDF of the sample +means.

    diff --git a/doc/pub/Regression/html/._Regression-bs061.html b/doc/pub/Regression/html/._Regression-bs061.html index e820b9387..b7395a27f 100644 --- a/doc/pub/Regression/html/._Regression-bs061.html +++ b/doc/pub/Regression/html/._Regression-bs061.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -381,28 +381,23 @@ MathJax.Hub.Config({

     

     

     

    - + -

    Statistics and sample variables

    +

    Statistics

    -In practical situations a sample is always of finite size. Let that -size be \( n \). The expectation value of a sample, the sample mean, is then defined as follows: +The probability density of obtaining a sample mean \( \bar x_n \) +is the product of probabilities of obtaining arbitrary values \( x_1, +x_2,\dots,x_n \) with the constraint that the mean of the set \( \{x_i\} \) +is \( \bar x_n \): $$ -\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots +\int p_X^{\phantom X}(x_n)\ +\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 $$ -The sample variance is: -$$ -\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 -$$ - -its square root being the standard deviation of the sample. The -sample covariance is: -$$ -\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) -$$ +And in particular we are interested in its variance \( \mathrm{var}(\overline X_n) \).

    diff --git a/doc/pub/Regression/html/._Regression-bs062.html b/doc/pub/Regression/html/._Regression-bs062.html index 4fda2d83b..34493f2e5 100644 --- a/doc/pub/Regression/html/._Regression-bs062.html +++ b/doc/pub/Regression/html/._Regression-bs062.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,21 +383,25 @@ MathJax.Hub.Config({ -

    Statistics, sample variance and covariance

    +

    Statistics, central limit theorem

    -Note that the sample variance is the sample covariance without the -cross terms. In a similar manner as the covariance in Eq. (12) is a measure of the correlation between -two stochastic variables, the above defined sample covariance is a -measure of the sequential correlation between succeeding measurements -of a sample. - -

    -These quantities, being known experimental values, differ -significantly from and must not be confused with the similarly named -quantities for stochastic variables, mean \( \mu_X \), variance \( \mathrm{var}(X) \) -and covariance \( \mathrm{cov}(X,Y) \). +It is generally not possible to express \( p_{\overline X_n}(x) \) in a +closed form given an arbitrary PDF \( p_X^{\phantom X} \) and a number +\( n \). But for the limit \( n\to\infty \) it is possible to make an +approximation. The very important result is called the central limit theorem. It tells us that as \( n \) goes to infinity, +\( p_{\overline X_n}(x) \) approaches a Gaussian distribution whose mean +and variance equal the true mean and variance, \( \mu_{X}^{\phantom X} \) +and \( \sigma_{X}^{2} \), respectively: +$$ +\begin{equation} +\lim_{n\to\infty} p_{\overline X_n}(x) = +\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} +e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} +\tag{13} +\end{equation} +$$

    diff --git a/doc/pub/Regression/html/._Regression-bs063.html b/doc/pub/Regression/html/._Regression-bs063.html index 4248aec4f..533907a24 100644 --- a/doc/pub/Regression/html/._Regression-bs063.html +++ b/doc/pub/Regression/html/._Regression-bs063.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,32 +383,29 @@ MathJax.Hub.Config({ -

    Statistics, law of large numbers

    +

    Statistics, more technicalities

    -The law of large numbers -states that as the size of our sample grows to infinity, the sample -mean approaches the true mean \( \mu_X^{\phantom X} \) of the chosen PDF: +The desired variance +\( \mathrm{var}(\overline X_n) \), i.e. the sample error squared +\( \mathrm{err}_X^2 \), is given by: $$ -\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} +\begin{equation} +\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} +\sum_{ij} \mathrm{cov}(X_i, X_j) +\tag{14} +\end{equation} $$ -The sample mean \( \bar{x}_n \) works therefore as an estimate of the true -mean \( \mu_X^{\phantom X} \). - -

    -What we need to find out is how good an approximation \( \bar{x}_n \) is to -\( \mu_X^{\phantom X} \). In any stochastic measurement, an estimated -mean is of no use to us without a measure of its error. A quantity -that tells us how well we can reproduce it in another experiment. We -are therefore interested in the PDF of the sample mean itself. Its -standard deviation will be a measure of the spread of sample means, -and we will simply call it the error of the sample mean, or -just sample error, and denote it by \( \mathrm{err}_X^{\phantom X} \). In -practice, we will only be able to produce an estimate of the -sample error since the exact value would require the knowledge of the -true PDFs behind, which we usually do not have. +We see now that in order to calculate the exact error of the sample +with the above expression, we would need the true means +\( \mu_{X_i}^{\phantom X} \) of the stochastic variables \( X_i \). To +calculate these requires that we know the true multivariate PDF of all +the \( X_i \). But this PDF is unknown to us, we have only got the measurements of +one sample. The best we can do is to let the sample itself be an +estimate of the PDF of each of the \( X_i \), estimating all properties of +\( X_i \) through the measurements of the sample.

    diff --git a/doc/pub/Regression/html/._Regression-bs064.html b/doc/pub/Regression/html/._Regression-bs064.html index 1c1b07735..71b4a66b1 100644 --- a/doc/pub/Regression/html/._Regression-bs064.html +++ b/doc/pub/Regression/html/._Regression-bs064.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,22 +383,27 @@ MathJax.Hub.Config({ -

    Statistics, more on sample error

    +

    Statistics

    -Let us first take a look at what happens to the sample error as the -size of the sample grows. In a sample, each of the measurements \( x_i \) -can be associated with its own stochastic variable \( X_i \). The -stochastic variable \( \overline X_n \) for the sample mean \( \bar{x}_n \) is -then just a linear combination, already familiar to us: +Our estimate of \( \mu_{X_i}^{\phantom X} \) is then the sample mean \( \bar x \) +itself, in accordance with the the central limit theorem: $$ -\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i +\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x $$ -All the coefficients are just equal \( 1/n \). The PDF of \( \overline X_n \), -denoted by \( p_{\overline X_n}(x) \) is the desired PDF of the sample -means. +Using \( \bar x \) in place of \( \mu_{X_i}^{\phantom X} \) we can give an +estimate of the covariance in Eq. (14) +$$ +\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, +$$ + +resulting in +$$ +\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) +$$

    diff --git a/doc/pub/Regression/html/._Regression-bs065.html b/doc/pub/Regression/html/._Regression-bs065.html index f779adecb..552e00de3 100644 --- a/doc/pub/Regression/html/._Regression-bs065.html +++ b/doc/pub/Regression/html/._Regression-bs065.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,21 +383,39 @@ MathJax.Hub.Config({ -

    Statistics

    +

    Statistics and sample variance

    -The probability density of obtaining a sample mean \( \bar x_n \) -is the product of probabilities of obtaining arbitrary values \( x_1, -x_2,\dots,x_n \) with the constraint that the mean of the set \( \{x_i\} \) -is \( \bar x_n \): +By the same procedure we can use the sample variance as an +estimate of the variance of any of the stochastic variables \( X_i \) $$ -p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots -\int p_X^{\phantom X}(x_n)\ -\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 +\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, $$ -And in particular we are interested in its variance \( \mathrm{var}(\overline X_n) \). +which is approximated as +$$ +\begin{equation} +\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) +\tag{15} +\end{equation} +$$ + +

    +Now we can calculate an estimate of the error +\( \mathrm{err}_X^{\phantom X} \) of the sample mean \( \bar x_n \): +$$ +\begin{align} +\mathrm{err}_X^2 +&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ +&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ +&=\frac{1}{n}\mathrm{cov}(x) +\tag{16} +\end{align} +$$ + +which is nothing but the sample covariance divided by the number of +measurements in the sample.

    diff --git a/doc/pub/Regression/html/._Regression-bs066.html b/doc/pub/Regression/html/._Regression-bs066.html index d12754001..89800e2fa 100644 --- a/doc/pub/Regression/html/._Regression-bs066.html +++ b/doc/pub/Regression/html/._Regression-bs066.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,25 +383,35 @@ MathJax.Hub.Config({ -

    Statistics, central limit theorem

    +

    Statistics, uncorrelated results

    -It is generally not possible to express \( p_{\overline X_n}(x) \) in a -closed form given an arbitrary PDF \( p_X^{\phantom X} \) and a number -\( n \). But for the limit \( n\to\infty \) it is possible to make an -approximation. The very important result is called the central limit theorem. It tells us that as \( n \) goes to infinity, -\( p_{\overline X_n}(x) \) approaches a Gaussian distribution whose mean -and variance equal the true mean and variance, \( \mu_{X}^{\phantom X} \) -and \( \sigma_{X}^{2} \), respectively: + +

    +In the special case that the measurements of the sample are +uncorrelated (equivalently the stochastic variables \( X_i \) are +uncorrelated) we have that the off-diagonal elements of the covariance +are zero. This gives the following estimate of the sample error: +$$ +\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = +\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +$$ + +resulting in $$ \begin{equation} -\lim_{n\to\infty} p_{\overline X_n}(x) = -\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} -e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} -\tag{19} +\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) +\tag{17} \end{equation} $$ + +where in the second step we have used Eq. (15). +The error of the sample is then just its standard deviation divided by +the square root of the number of measurements the sample contains. +This is a very useful formula which is easy to compute. It acts as a +first approximation to the error, but in numerical experiments, we +cannot overlook the always present correlations.

    diff --git a/doc/pub/Regression/html/._Regression-bs067.html b/doc/pub/Regression/html/._Regression-bs067.html index ba2cdc20b..f94ac7265 100644 --- a/doc/pub/Regression/html/._Regression-bs067.html +++ b/doc/pub/Regression/html/._Regression-bs067.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,29 +383,29 @@ MathJax.Hub.Config({ -

    Statistics, more technicalities

    +

    Statistics, computations

    -The desired variance -\( \mathrm{var}(\overline X_n) \), i.e. the sample error squared -\( \mathrm{err}_X^2 \), is given by: +For computational purposes one usually splits up the estimate of +\( \mathrm{err}_X^2 \), given by Eq. (16), into two +parts +$$ +\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +$$ + +which equals $$ \begin{equation} -\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} -\sum_{ij} \mathrm{cov}(X_i, X_j) -\tag{20} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) +\tag{18} \end{equation} $$ -We see now that in order to calculate the exact error of the sample -with the above expression, we would need the true means -\( \mu_{X_i}^{\phantom X} \) of the stochastic variables \( X_i \). To -calculate these requires that we know the true multivariate PDF of all -the \( X_i \). But this PDF is unknown to us, we have only got the measurements of -one sample. The best we can do is to let the sample itself be an -estimate of the PDF of each of the \( X_i \), estimating all properties of -\( X_i \) through the measurements of the sample. +The first term is the same as the error in the uncorrelated case, +Eq. (17). This means that the second +term accounts for the error correction due to correlation between the +measurements. For uncorrelated measurements this second term is zero.

    diff --git a/doc/pub/Regression/html/._Regression-bs068.html b/doc/pub/Regression/html/._Regression-bs068.html index a53e75292..97ff76be5 100644 --- a/doc/pub/Regression/html/._Regression-bs068.html +++ b/doc/pub/Regression/html/._Regression-bs068.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,27 +383,22 @@ MathJax.Hub.Config({ -

    Statistics

    +

    Statistics, more on computations of errors

    -Our estimate of \( \mu_{X_i}^{\phantom X} \) is then the sample mean \( \bar x \) -itself, in accordance with the the central limit theorem: +Computationally the uncorrelated first term is much easier to treat +efficiently than the second. $$ -\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x +\mathrm{var}(x) = \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)^2 = +\left(\frac{1}{n}\sum_{k=1}^n x_k^2\right) - \bar x_n^2 $$ -Using \( \bar x \) in place of \( \mu_{X_i}^{\phantom X} \) we can give an -estimate of the covariance in Eq. (20) -$$ -\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle -\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, -$$ - -resulting in -$$ -\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) -$$ +We just accumulate separately the values \( x^2 \) and \( x \) for every +measurement \( x \) we receive. The correlation term, though, has to be +calculated at the end of the experiment since we need all the +measurements to calculate the cross terms. Therefore, all measurements +have to be stored throughout the experiment.

    diff --git a/doc/pub/Regression/html/._Regression-bs069.html b/doc/pub/Regression/html/._Regression-bs069.html index 45900b23e..f7e050e27 100644 --- a/doc/pub/Regression/html/._Regression-bs069.html +++ b/doc/pub/Regression/html/._Regression-bs069.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,39 +383,33 @@ MathJax.Hub.Config({ -

    Statistics and sample variance

    +

    Statistics, wrapping up 1

    -By the same procedure we can use the sample variance as an -estimate of the variance of any of the stochastic variables \( X_i \) +Let us analyze the problem by splitting up the correlation term into +partial sums of the form: $$ -\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, +f_d = \frac{1}{n-d}\sum_{k=1}^{n-d}(x_k - \bar x_n)(x_{k+d} - \bar x_n) $$ -which is approximated as +The correlation term of the error can now be rewritten in terms of +\( f_d \) $$ -\begin{equation} -\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) -\tag{21} -\end{equation} +\frac{2}{n}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) = +2\sum_{d=1}^{n-1} f_d $$ -

    -Now we can calculate an estimate of the error -\( \mathrm{err}_X^{\phantom X} \) of the sample mean \( \bar x_n \): +The value of \( f_d \) reflects the correlation between measurements +separated by the distance \( d \) in the sample samples. Notice that for +\( d=0 \), \( f \) is just the sample variance, \( \mathrm{var}(x) \). If we divide \( f_d \) +by \( \mathrm{var}(x) \), we arrive at the so called autocorrelation function $$ -\begin{align} -\mathrm{err}_X^2 -&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ -&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ -&=\frac{1}{n}\mathrm{cov}(x) -\tag{22} -\end{align} +\kappa_d = \frac{f_d}{\mathrm{var}(x)} $$ -which is nothing but the sample covariance divided by the number of -measurements in the sample. +which gives us a useful measure of pairwise correlations +starting always at \( 1 \) for \( d=0 \).

    diff --git a/doc/pub/Regression/html/._Regression-bs070.html b/doc/pub/Regression/html/._Regression-bs070.html index b5c9f9812..bd0dcd1f0 100644 --- a/doc/pub/Regression/html/._Regression-bs070.html +++ b/doc/pub/Regression/html/._Regression-bs070.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,35 +383,33 @@ MathJax.Hub.Config({ -

    Statistics, uncorrelated results

    +

    Statistics, final expression

    - -

    -In the special case that the measurements of the sample are -uncorrelated (equivalently the stochastic variables \( X_i \) are -uncorrelated) we have that the off-diagonal elements of the covariance -are zero. This gives the following estimate of the sample error: +The sample error (see eq. (18)) can now be +written in terms of the autocorrelation function: $$ -\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = -\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +\begin{align} +\mathrm{err}_X^2 &= +\frac{1}{n}\mathrm{var}(x)+\frac{2}{n}\cdot\mathrm{var}(x)\sum_{d=1}^{n-1} +\frac{f_d}{\mathrm{var}(x)}\nonumber\\ &=& +\left(1+2\sum_{d=1}^{n-1}\kappa_d\right)\frac{1}{n}\mathrm{var}(x)\nonumber\\ +&=\frac{\tau}{n}\cdot\mathrm{var}(x) +\tag{19} +\end{align} $$ -resulting in +and we see that \( \mathrm{err}_X \) can be expressed in terms the +uncorrelated sample variance times a correction factor \( \tau \) which +accounts for the correlation between measurements. We call this +correction factor the autocorrelation time: $$ \begin{equation} -\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) -\tag{23} +\tau = 1+2\sum_{d=1}^{n-1}\kappa_d +\tag{20} \end{equation} $$ - -where in the second step we have used Eq. (21). -The error of the sample is then just its standard deviation divided by -the square root of the number of measurements the sample contains. -This is a very useful formula which is easy to compute. It acts as a -first approximation to the error, but in numerical experiments, we -cannot overlook the always present correlations.

    diff --git a/doc/pub/Regression/html/._Regression-bs071.html b/doc/pub/Regression/html/._Regression-bs071.html index adf7712e5..c8cada5d7 100644 --- a/doc/pub/Regression/html/._Regression-bs071.html +++ b/doc/pub/Regression/html/._Regression-bs071.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,29 +383,25 @@ MathJax.Hub.Config({ -

    Statistics, computations

    +

    Statistics, effective number of correlations

    -For computational purposes one usually splits up the estimate of -\( \mathrm{err}_X^2 \), given by Eq. (22), into two -parts +For a correlation free experiment, \( \tau \) +equals 1. From the point of view of +eq. (19) we can interpret a sequential +correlation as an effective reduction of the number of measurements by +a factor \( \tau \). The effective number of measurements becomes: $$ -\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +n_\mathrm{eff} = \frac{n}{\tau} $$ -which equals -$$ -\begin{equation} -\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) -\tag{24} -\end{equation} -$$ - -The first term is the same as the error in the uncorrelated case, -Eq. (23). This means that the second -term accounts for the error correction due to correlation between the -measurements. For uncorrelated measurements this second term is zero. +To neglect the autocorrelation time \( \tau \) will always cause our +simple uncorrelated estimate of \( \mathrm{err}_X^2\approx \mathrm{var}(x)/n \) to +be less than the true sample error. The estimate of the error will be +too good. On the other hand, the calculation of the full +autocorrelation time poses an efficiency problem if the set of +measurements is very large.

    diff --git a/doc/pub/Regression/html/._Regression-bs072.html b/doc/pub/Regression/html/._Regression-bs072.html index aa8990750..07713ece8 100644 --- a/doc/pub/Regression/html/._Regression-bs072.html +++ b/doc/pub/Regression/html/._Regression-bs072.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({
  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -381,27 +381,42 @@ MathJax.Hub.Config({

     

     

     

    - + -

    Statistics, more on computations of errors

    -
    -
    -

    -Computationally the uncorrelated first term is much easier to treat -efficiently than the second. +

    Linking the regression analysis with a statistical interpretation

    + +

    +Finally, we are going to discuss several statistical properties which can be obtained in terms of analytical expressions. +The +advantage of doing linear regression is that we actually end up with +analytical expressions for several statistical quantities. +Standard least squares and Ridge regression allow us to +derive quantities like the variance and other expectation values in a +rather straightforward way. + +

    +It is assumed that \( \varepsilon_i +\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are +independent, i.e.: $$ -\mathrm{var}(x) = \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)^2 = -\left(\frac{1}{n}\sum_{k=1}^n x_k^2\right) - \bar x_n^2 +\begin{align*} +\mbox{Cov}(\varepsilon_{i_1}, +\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} +& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. +\end{align*} $$ -We just accumulate separately the values \( x^2 \) and \( x \) for every -measurement \( x \) we receive. The correlation term, though, has to be -calculated at the end of the experiment since we need all the -measurements to calculate the cross terms. Therefore, all measurements -have to be stored throughout the experiment. -

    -
    +The randomness of \( \varepsilon_i \) implies that +\( \mathbf{y}_i \) is also a random variable. In particular, +\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim +\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a +non-random scalar. To specify the parameters of the distribution of +\( \mathbf{y}_i \) we need to calculate its first two moments. +

    +Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The +notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the +row number \( i \) and perform a sum over all values \( p \).

    diff --git a/doc/pub/Regression/html/._Regression-bs073.html b/doc/pub/Regression/html/._Regression-bs073.html index 541bac035..ebcb6cf6c 100644 --- a/doc/pub/Regression/html/._Regression-bs073.html +++ b/doc/pub/Regression/html/._Regression-bs073.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,37 +383,23 @@ MathJax.Hub.Config({ -

    Statistics, wrapping up 1

    -
    -
    -

    -Let us analyze the problem by splitting up the correlation term into -partial sums of the form: +

    Assumptions made

    + +

    +The assumption we have made here can be summarized as (and this is going to useful when we discuss the bias-variance trade off) +that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) +which describes our data $$ -f_d = \frac{1}{n-d}\sum_{k=1}^{n-d}(x_k - \bar x_n)(x_{k+d} - \bar x_n) +\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} $$ -The correlation term of the error can now be rewritten in terms of -\( f_d \) +

    +We approximate this function with our model from the solution of the linear regression equations, that is our +function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with $$ -\frac{2}{n}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) = -2\sum_{d=1}^{n-1} f_d +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. $$ -The value of \( f_d \) reflects the correlation between measurements -separated by the distance \( d \) in the sample samples. Notice that for -\( d=0 \), \( f \) is just the sample variance, \( \mathrm{var}(x) \). If we divide \( f_d \) -by \( \mathrm{var}(x) \), we arrive at the so called autocorrelation function -$$ -\kappa_d = \frac{f_d}{\mathrm{var}(x)} -$$ - -which gives us a useful measure of pairwise correlations -starting always at \( 1 \) for \( d=0 \). -

    -
    - -

    diff --git a/doc/pub/Regression/html/._Regression-bs074.html b/doc/pub/Regression/html/._Regression-bs074.html index 4fe97e6f0..0732b2c11 100644 --- a/doc/pub/Regression/html/._Regression-bs074.html +++ b/doc/pub/Regression/html/._Regression-bs074.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,36 +383,37 @@ MathJax.Hub.Config({ -

    Statistics, final expression

    -
    -
    -

    -The sample error (see eq. (24)) can now be -written in terms of the autocorrelation function: +

    Expectation value and variance

    + +

    +We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \) $$ -\begin{align} -\mathrm{err}_X^2 &= -\frac{1}{n}\mathrm{var}(x)+\frac{2}{n}\cdot\mathrm{var}(x)\sum_{d=1}^{n-1} -\frac{f_d}{\mathrm{var}(x)}\nonumber\\ &=& -\left(1+2\sum_{d=1}^{n-1}\kappa_d\right)\frac{1}{n}\mathrm{var}(x)\nonumber\\ -&=\frac{\tau}{n}\cdot\mathrm{var}(x) -\tag{25} -\end{align} +\begin{align*} +\mathbb{E}(y_i) & = +\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\end{align*} $$ -and we see that \( \mathrm{err}_X \) can be expressed in terms the -uncorrelated sample variance times a correction factor \( \tau \) which -accounts for the correlation between measurements. We call this -correction factor the autocorrelation time: +while +its variance is $$ -\begin{equation} -\tau = 1+2\sum_{d=1}^{n-1}\kappa_d -\tag{26} -\end{equation} +\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i +- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - +[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, +\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 +\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, +\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. +\end{align*} $$ -

    -
    +Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).

    diff --git a/doc/pub/Regression/html/._Regression-bs075.html b/doc/pub/Regression/html/._Regression-bs075.html index faf8babc4..0af6f2aa2 100644 --- a/doc/pub/Regression/html/._Regression-bs075.html +++ b/doc/pub/Regression/html/._Regression-bs075.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,28 +383,87 @@ MathJax.Hub.Config({ -

    Statistics, effective number of correlations

    -
    -
    -

    -For a correlation free experiment, \( \tau \) -equals 1. From the point of view of -eq. (25) we can interpret a sequential -correlation as an effective reduction of the number of measurements by -a factor \( \tau \). The effective number of measurements becomes: +

    Expectation value and variance for \( \boldsymbol{\beta} \)

    + +

    +With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value $$ -n_\mathrm{eff} = \frac{n}{\tau} +\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. $$ -To neglect the autocorrelation time \( \tau \) will always cause our -simple uncorrelated estimate of \( \mathrm{err}_X^2\approx \mathrm{var}(x)/n \) to -be less than the true sample error. The estimate of the error will be -too good. On the other hand, the calculation of the full -autocorrelation time poses an efficiency problem if the set of -measurements is very large. -

    -
    +This means that the estimator of the regression parameters is unbiased. +

    +We can also calculate the variance + +

    +The variance of \( \boldsymbol{\beta} \) is +$$ +\begin{eqnarray*} +\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} +\\ +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} +\\ +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% \\ +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T +\\ +& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, +\end{eqnarray*} +$$ + +

    +where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 +\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the +variance of the estimate of the \( j \)-th regression coefficient: +\( \hat{\sigma}^2 (\hat{\beta}_j ) = \hat{\sigma}^2 \sqrt{ +[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} } \). This may be used to +construct a confidence interval for the estimates. + +

    +In a similar way, we cna obtain analytical expressions for say the +expectation values of the parameters \( \boldsymbol{\beta} \) and their variance +when we employ Ridge regression, and thereby a confidence interval. + +

    +It is rather straightforward to show that +$$ +\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +$$ + +We see clearly that +\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. + +

    +We can also compute the variance as + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +$$ + +and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. + +

    +With this, we can compute the difference + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +$$ + +The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.

    diff --git a/doc/pub/Regression/html/._Regression-bs076.html b/doc/pub/Regression/html/._Regression-bs076.html index a2a888cab..435769825 100644 --- a/doc/pub/Regression/html/._Regression-bs076.html +++ b/doc/pub/Regression/html/._Regression-bs076.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,40 +383,36 @@ MathJax.Hub.Config({ -

    Linking the regression analysis with a statistical interpretation

    +

    Cross-validation

    -Finally, we are going to discuss several statistical properties which can be obtained in terms of analytical expressions. -The -advantage of doing linear regression is that we actually end up with -analytical expressions for several statistical quantities. -Standard least squares and Ridge regression allow us to -derive quantities like the variance and other expectation values in a -rather straightforward way. +Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e. the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand.

    -It is assumed that \( \varepsilon_i -\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are -independent, i.e.: -$$ -\begin{align*} -\mbox{Cov}(\varepsilon_{i_1}, -\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} -& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. -\end{align*} -$$ - -The randomness of \( \varepsilon_i \) implies that -\( \mathbf{y}_i \) is also a random variable. In particular, -\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim -\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a -non-random scalar. To specify the parameters of the distribution of -\( \mathbf{y}_i \) we need to calculate its first two moments. - -

    -Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The -notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the -row number \( i \) and perform a sum over all values \( p \). +The setting of original and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the +training set, plays the role of original data on which the model is +built. The second of these data sets, called the test set, plays the +role of the novel data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of \( \lambda \) using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected.

    diff --git a/doc/pub/Regression/html/._Regression-bs077.html b/doc/pub/Regression/html/._Regression-bs077.html index 1a192e2bd..61ce3a51d 100644 --- a/doc/pub/Regression/html/._Regression-bs077.html +++ b/doc/pub/Regression/html/._Regression-bs077.html @@ -111,112 +111,112 @@ Automatically generated HTML file from DocOnce source ('The Boston housing data example', 2, None, '___sec27'), ('Housing data, the code', 2, None, '___sec28'), ('The singular value decomposition', 2, None, '___sec29'), - ('The Ising model', 2, None, '___sec30'), - ('Reformulating the problem to suit regression', - 2, - None, - '___sec31'), - ('Linear regression', 2, None, '___sec32'), - ('Singular Value decomposition', 2, None, '___sec33'), - ('Linear Regression Problems', 2, None, '___sec34'), - ('Fixing the singularity', 2, None, '___sec35'), - ('Basic math of the SVD', 2, None, '___sec36'), - ('The SVD, a Fantastic Algorithm', 2, None, '___sec37'), - ('Another Example', 2, None, '___sec38'), - ('Economy-size SVD', 2, None, '___sec39'), - ('Mathematical Properties', 2, None, '___sec40'), - ('Ridge and LASSO Regression', 2, None, '___sec41'), - ('More on Ridge Regression', 2, None, '___sec42'), - ('Interpreting the Ridge results', 2, None, '___sec43'), - ('More interpretations', 2, None, '___sec44'), - ('Where are we going?', 2, None, '___sec45'), - ('Resampling methods', 2, None, '___sec46'), + ('Linear Regression Problems', 2, None, '___sec30'), + ('Fixing the singularity', 2, None, '___sec31'), + ('Basic math of the SVD', 2, None, '___sec32'), + ('The SVD, a Fantastic Algorithm', 2, None, '___sec33'), + ('Another Example', 2, None, '___sec34'), + ('Economy-size SVD', 2, None, '___sec35'), + ('Mathematical Properties', 2, None, '___sec36'), + ('Ridge and LASSO Regression', 2, None, '___sec37'), + ('More on Ridge Regression', 2, None, '___sec38'), + ('Interpreting the Ridge results', 2, None, '___sec39'), + ('More interpretations', 2, None, '___sec40'), + ('Where are we going?', 2, None, '___sec41'), + ('Resampling methods', 2, None, '___sec42'), ('Resampling approaches can be computationally expensive', 2, None, - '___sec47'), - ('Why resampling methods ?', 2, None, '___sec48'), - ('Statistical analysis', 2, None, '___sec49'), - ('Statistics', 2, None, '___sec50'), - ('Statistics, moments', 2, None, '___sec51'), - ('Statistics, central moments', 2, None, '___sec52'), - ('Statistics, covariance', 2, None, '___sec53'), - ('Statistics, more covariance', 2, None, '___sec54'), - ('Covariance example', 2, None, '___sec55'), - ('Covariance in numpy', 2, None, '___sec56'), - ('Statistics, independent variables', 2, None, '___sec57'), - ('Statistics, more variance', 2, None, '___sec58'), - ('Statistics and stochastic processes', 2, None, '___sec59'), - ('Statistics and sample variables', 2, None, '___sec60'), + '___sec43'), + ('Why resampling methods ?', 2, None, '___sec44'), + ('Statistical analysis', 2, None, '___sec45'), + ('Statistics', 2, None, '___sec46'), + ('Statistics, moments', 2, None, '___sec47'), + ('Statistics, central moments', 2, None, '___sec48'), + ('Statistics, covariance', 2, None, '___sec49'), + ('Statistics, more covariance', 2, None, '___sec50'), + ('Covariance example', 2, None, '___sec51'), + ('Covariance in numpy', 2, None, '___sec52'), + ('Statistics, independent variables', 2, None, '___sec53'), + ('Statistics, more variance', 2, None, '___sec54'), + ('Statistics and stochastic processes', 2, None, '___sec55'), + ('Statistics and sample variables', 2, None, '___sec56'), ('Statistics, sample variance and covariance', 2, None, - '___sec61'), - ('Statistics, law of large numbers', 2, None, '___sec62'), - ('Statistics, more on sample error', 2, None, '___sec63'), - ('Statistics', 2, None, '___sec64'), - ('Statistics, central limit theorem', 2, None, '___sec65'), - ('Statistics, more technicalities', 2, None, '___sec66'), - ('Statistics', 2, None, '___sec67'), - ('Statistics and sample variance', 2, None, '___sec68'), - ('Statistics, uncorrelated results', 2, None, '___sec69'), - ('Statistics, computations', 2, None, '___sec70'), + '___sec57'), + ('Statistics, law of large numbers', 2, None, '___sec58'), + ('Statistics, more on sample error', 2, None, '___sec59'), + ('Statistics', 2, None, '___sec60'), + ('Statistics, central limit theorem', 2, None, '___sec61'), + ('Statistics, more technicalities', 2, None, '___sec62'), + ('Statistics', 2, None, '___sec63'), + ('Statistics and sample variance', 2, None, '___sec64'), + ('Statistics, uncorrelated results', 2, None, '___sec65'), + ('Statistics, computations', 2, None, '___sec66'), ('Statistics, more on computations of errors', 2, None, - '___sec71'), - ('Statistics, wrapping up 1', 2, None, '___sec72'), - ('Statistics, final expression', 2, None, '___sec73'), + '___sec67'), + ('Statistics, wrapping up 1', 2, None, '___sec68'), + ('Statistics, final expression', 2, None, '___sec69'), ('Statistics, effective number of correlations', 2, None, - '___sec74'), + '___sec70'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, None, - '___sec75'), - ('Assumptions made', 2, None, '___sec76'), - ('Expectation value and variance', 2, None, '___sec77'), + '___sec71'), + ('Assumptions made', 2, None, '___sec72'), + ('Expectation value and variance', 2, None, '___sec73'), ('Expectation value and variance for $\\boldsymbol{\\beta}$', 2, None, - '___sec78'), - ('Cross-validation', 2, None, '___sec79'), - ('Computationally expensive', 2, None, '___sec80'), - ('Various steps in cross-validation', 2, None, '___sec81'), + '___sec74'), + ('Cross-validation', 2, None, '___sec75'), + ('Computationally expensive', 2, None, '___sec76'), + ('Various steps in cross-validation', 2, None, '___sec77'), ('How to set up the cross-validation for Ridge and/or Lasso', 2, None, - '___sec82'), + '___sec78'), ('Resampling methods: Jackknife and Bootstrap', 2, None, - '___sec83'), - ('Resampling methods: Jackknife', 2, None, '___sec84'), - ('Jackknife code example', 2, None, '___sec85'), - ('Resampling methods: Bootstrap', 2, None, '___sec86'), - ('Resampling methods: Bootstrap background', 2, None, '___sec87'), + '___sec79'), + ('Resampling methods: Jackknife', 2, None, '___sec80'), + ('Jackknife code example', 2, None, '___sec81'), + ('Resampling methods: Bootstrap', 2, None, '___sec82'), + ('Resampling methods: Bootstrap background', 2, None, '___sec83'), ('Resampling methods: More Bootstrap background', 2, None, - '___sec88'), - ('Resampling methods: Bootstrap approach', 2, None, '___sec89'), - ('Resampling methods: Bootstrap steps', 2, None, '___sec90'), - ('Code example for the Bootstrap method', 2, None, '___sec91'), + '___sec84'), + ('Resampling methods: Bootstrap approach', 2, None, '___sec85'), + ('Resampling methods: Bootstrap steps', 2, None, '___sec86'), + ('Code example for the Bootstrap method', 2, None, '___sec87'), ('Code Example for Cross-validation and $k$-fold ' 'Cross-validation', 2, None, - '___sec92'), - ('The bias-variance tradeoff', 2, None, '___sec93'), - ('Example code for Bias-Variance tradeoff', 2, None, '___sec94'), - ('Understanding what happens', 2, None, '___sec95'), - ('Summing up', 2, None, '___sec96'), + '___sec88'), + ('The bias-variance tradeoff', 2, None, '___sec89'), + ('Example code for Bias-Variance tradeoff', 2, None, '___sec90'), + ('Understanding what happens', 2, None, '___sec91'), + ('Summing up', 2, None, '___sec92'), ("Another Example rom Scikit-Learn's Repository", 2, None, - '___sec97'), + '___sec93'), + ('The Ising model', 2, None, '___sec94'), + ('Reformulating the problem to suit regression', + 2, + None, + '___sec95'), + ('Linear regression', 2, None, '___sec96'), + ('Singular Value decomposition', 2, None, '___sec97'), ('The one-dimensional Ising model', 2, None, '___sec98'), ('Ridge regression', 2, None, '___sec99'), ('LASSO regression', 2, None, '___sec100'), @@ -295,74 +295,74 @@ MathJax.Hub.Config({

  • The Boston housing data example
  • Housing data, the code
  • The singular value decomposition
  • -
  • The Ising model
  • -
  • Reformulating the problem to suit regression
  • -
  • Linear regression
  • -
  • Singular Value decomposition
  • -
  • Linear Regression Problems
  • -
  • Fixing the singularity
  • -
  • Basic math of the SVD
  • -
  • The SVD, a Fantastic Algorithm
  • -
  • Another Example
  • -
  • Economy-size SVD
  • -
  • Mathematical Properties
  • -
  • Ridge and LASSO Regression
  • -
  • More on Ridge Regression
  • -
  • Interpreting the Ridge results
  • -
  • More interpretations
  • -
  • Where are we going?
  • -
  • Resampling methods
  • -
  • Resampling approaches can be computationally expensive
  • -
  • Why resampling methods ?
  • -
  • Statistical analysis
  • -
  • Statistics
  • -
  • Statistics, moments
  • -
  • Statistics, central moments
  • -
  • Statistics, covariance
  • -
  • Statistics, more covariance
  • -
  • Covariance example
  • -
  • Covariance in numpy
  • -
  • Statistics, independent variables
  • -
  • Statistics, more variance
  • -
  • Statistics and stochastic processes
  • -
  • Statistics and sample variables
  • -
  • Statistics, sample variance and covariance
  • -
  • Statistics, law of large numbers
  • -
  • Statistics, more on sample error
  • -
  • Statistics
  • -
  • Statistics, central limit theorem
  • -
  • Statistics, more technicalities
  • -
  • Statistics
  • -
  • Statistics and sample variance
  • -
  • Statistics, uncorrelated results
  • -
  • Statistics, computations
  • -
  • Statistics, more on computations of errors
  • -
  • Statistics, wrapping up 1
  • -
  • Statistics, final expression
  • -
  • Statistics, effective number of correlations
  • -
  • Linking the regression analysis with a statistical interpretation
  • -
  • Assumptions made
  • -
  • Expectation value and variance
  • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • -
  • Cross-validation
  • -
  • Computationally expensive
  • -
  • Various steps in cross-validation
  • -
  • How to set up the cross-validation for Ridge and/or Lasso
  • -
  • Resampling methods: Jackknife and Bootstrap
  • -
  • Resampling methods: Jackknife
  • -
  • Jackknife code example
  • -
  • Resampling methods: Bootstrap
  • -
  • Resampling methods: Bootstrap background
  • -
  • Resampling methods: More Bootstrap background
  • -
  • Resampling methods: Bootstrap approach
  • -
  • Resampling methods: Bootstrap steps
  • -
  • Code example for the Bootstrap method
  • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • -
  • The bias-variance tradeoff
  • -
  • Example code for Bias-Variance tradeoff
  • -
  • Understanding what happens
  • -
  • Summing up
  • -
  • Another Example rom Scikit-Learn's Repository
  • +
  • Linear Regression Problems
  • +
  • Fixing the singularity
  • +
  • Basic math of the SVD
  • +
  • The SVD, a Fantastic Algorithm
  • +
  • Another Example
  • +
  • Economy-size SVD
  • +
  • Mathematical Properties
  • +
  • Ridge and LASSO Regression
  • +
  • More on Ridge Regression
  • +
  • Interpreting the Ridge results
  • +
  • More interpretations
  • +
  • Where are we going?
  • +
  • Resampling methods
  • +
  • Resampling approaches can be computationally expensive
  • +
  • Why resampling methods ?
  • +
  • Statistical analysis
  • +
  • Statistics
  • +
  • Statistics, moments
  • +
  • Statistics, central moments
  • +
  • Statistics, covariance
  • +
  • Statistics, more covariance
  • +
  • Covariance example
  • +
  • Covariance in numpy
  • +
  • Statistics, independent variables
  • +
  • Statistics, more variance
  • +
  • Statistics and stochastic processes
  • +
  • Statistics and sample variables
  • +
  • Statistics, sample variance and covariance
  • +
  • Statistics, law of large numbers
  • +
  • Statistics, more on sample error
  • +
  • Statistics
  • +
  • Statistics, central limit theorem
  • +
  • Statistics, more technicalities
  • +
  • Statistics
  • +
  • Statistics and sample variance
  • +
  • Statistics, uncorrelated results
  • +
  • Statistics, computations
  • +
  • Statistics, more on computations of errors
  • +
  • Statistics, wrapping up 1
  • +
  • Statistics, final expression
  • +
  • Statistics, effective number of correlations
  • +
  • Linking the regression analysis with a statistical interpretation
  • +
  • Assumptions made
  • +
  • Expectation value and variance
  • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
  • +
  • Cross-validation
  • +
  • Computationally expensive
  • +
  • Various steps in cross-validation
  • +
  • How to set up the cross-validation for Ridge and/or Lasso
  • +
  • Resampling methods: Jackknife and Bootstrap
  • +
  • Resampling methods: Jackknife
  • +
  • Jackknife code example
  • +
  • Resampling methods: Bootstrap
  • +
  • Resampling methods: Bootstrap background
  • +
  • Resampling methods: More Bootstrap background
  • +
  • Resampling methods: Bootstrap approach
  • +
  • Resampling methods: Bootstrap steps
  • +
  • Code example for the Bootstrap method
  • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
  • +
  • The bias-variance tradeoff
  • +
  • Example code for Bias-Variance tradeoff
  • +
  • Understanding what happens
  • +
  • Summing up
  • +
  • Another Example rom Scikit-Learn's Repository
  • +
  • The Ising model
  • +
  • Reformulating the problem to suit regression
  • +
  • Linear regression
  • +
  • Singular Value decomposition
  • The one-dimensional Ising model
  • Ridge regression
  • LASSO regression
  • @@ -383,24 +383,16 @@ MathJax.Hub.Config({ -

    Assumptions made

    +

    Computationally expensive

    -The assumption we have made here can be summarized as (and this is going to useful when we discuss the bias-variance trade off) -that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) -which describes our data -$$ -\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} -$$ +The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: -

    -We approximate this function with our model from the solution of the linear regression equations, that is our -function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with -$$ -\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. -$$ +

    -