diff --git a/doc/pub/week36/html/._week36-bs000.html b/doc/pub/week36/html/._week36-bs000.html index c79950484..12be7d203 100644 --- a/doc/pub/week36/html/._week36-bs000.html +++ b/doc/pub/week36/html/._week36-bs000.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
  • Plans for week 36
    • -
  • Thursday September 8
    • -
  • Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
    • -
  • Linear Regression and the SVD
    • -
  • What does it mean?
    • -
  • And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
    • -
  • Code for SVD and Inversion of Matrices
    • -
  • Inverse of Rectangular Matrix
    • -
  • Ridge and LASSO Regression
    • -
  • From OLS to Ridge and Lasso
    • -
  • Deriving the Ridge Regression Equations
    • -
  • Note on Scikit-Learn
    • -
  • Comparison with OLS
    • -
  • SVD analysis
    • -
  • Interpreting the Ridge results
    • -
  • More interpretations
    • -
  • Deriving the Lasso Regression Equations
    • -
  • Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
    • -
  • Ridge Regression
    • -
  • Lasso Regression
    • -
  • Yet another Example
    • -
  • The OLS case
    • -
  • The Ridge case
    • -
  • Writing the Cost Function
    • -
  • Lasso case
    • -
  • The first Case
    • -
  • Simple code for solving the above problem
    • -
  • With Lasso Regression
    • -
  • Another Example, now with a polynomial fit
    • -
  • To think about, first part
    • -
  • More thinking
    • -
  • Still thinking
    • -
  • What does centering (subtracting the mean values) mean mathematically?
    • -
  • Further Manipulations
    • -
  • Wrapping it up
    • -
  • Linear Regression code, Intercept handling first
    • -
  • Code Examples
    • -
  • Taking out the mean
    • -
  • Friday September 9
    • -
  • Linking the regression analysis with a statistical interpretation
    • -
  • Assumptions made
    • -
  • Expectation value and variance
    • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
    • -
  • Deriving OLS from a probability distribution
    • -
  • Independent and Identically Distrubuted (iid)
    • -
  • Maximum Likelihood Estimation (MLE)
    • -
  • A new Cost Function
    • -
  • More basic Statistics and Bayes' theorem
    • -
  • Marginal Probability
    • -
  • Conditional Probability
    • -
  • Bayes' Theorem
    • -
  • Interpretations of Bayes' Theorem
    • -
  • Example of Usage of Bayes' theorem
    • -
  • Doing it correctly
    • -
  • Bayes' Theorem and Ridge and Lasso Regression
    • -
  • Test Function for what happens with OLS, Ridge and Lasso
    • -
  • Invoking Bayes' theorem
    • -
  • Ridge and Bayes
    • -
  • Lasso and Bayes
    • -
  • Exercise 1: mean values and variances in linear regression
    • -
  • Exercise 2: Adding Ridge and Lasso Regression
    • +
  • Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
    • +
  • Linear Regression and the SVD
    • +
  • What does it mean?
    • +
  • And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
    • +
  • Code for SVD and Inversion of Matrices
    • +
  • Inverse of Rectangular Matrix
    • +
  • Ridge and LASSO Regression
    • +
  • From OLS to Ridge and Lasso
    • +
  • Deriving the Ridge Regression Equations
    • +
  • Note on Scikit-Learn
    • +
  • Comparison with OLS
    • +
  • SVD analysis
    • +
  • Interpreting the Ridge results
    • +
  • More interpretations
    • +
  • Deriving the Lasso Regression Equations
    • +
  • Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
    • +
  • Ridge Regression
    • +
  • Lasso Regression
    • +
  • Yet another Example
    • +
  • The OLS case
    • +
  • The Ridge case
    • +
  • Writing the Cost Function
    • +
  • Lasso case
    • +
  • The first Case
    • +
  • Simple code for solving the above problem
    • +
  • With Lasso Regression
    • +
  • Another Example, now with a polynomial fit
    • +
  • To think about, first part
    • +
  • More thinking
    • +
  • Still thinking
    • +
  • What does centering (subtracting the mean values) mean mathematically?
    • +
  • Further Manipulations
    • +
  • Wrapping it up
    • +
  • Linear Regression code, Intercept handling first
    • +
  • Code Examples
    • +
  • Taking out the mean
    • +
  • Friday September 9
    • +
  • Linking the regression analysis with a statistical interpretation
    • +
  • Assumptions made
    • +
  • Expectation value and variance
    • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
    • +
  • Deriving OLS from a probability distribution
    • +
  • Independent and Identically Distrubuted (iid)
    • +
  • Maximum Likelihood Estimation (MLE)
    • +
  • A new Cost Function
    • +
  • More basic Statistics and Bayes' theorem
    • +
  • Marginal Probability
    • +
  • Conditional Probability
    • +
  • Bayes' Theorem
    • +
  • Interpretations of Bayes' Theorem
    • +
  • Example of Usage of Bayes' theorem
    • +
  • Doing it correctly
    • +
  • Bayes' Theorem and Ridge and Lasso Regression
    • +
  • Test Function for what happens with OLS, Ridge and Lasso
    • +
  • Invoking Bayes' theorem
    • +
  • Ridge and Bayes
    • +
  • Lasso and Bayes
    • +
  • Exercise 1: mean values and variances in linear regression
    • +
  • Exercise 2: Adding Ridge and Lasso Regression
    • @@ -315,7 +313,7 @@ MathJax.Hub.Config({
    -

    Sep 9, 2022

    +

    May 29, 2023


    @@ -340,7 +338,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 60
    • +
  • 59
  • »
    • @@ -354,7 +352,7 @@ MathJax.Hub.Config({ -->
    - © 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    diff --git a/doc/pub/week36/html/._week36-bs001.html b/doc/pub/week36/html/._week36-bs001.html index 0f24a7bb5..d83f9eb68 100644 --- a/doc/pub/week36/html/._week36-bs001.html +++ b/doc/pub/week36/html/._week36-bs001.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
  • Plans for week 36
    • -
  • Thursday September 8
    • -
  • Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
    • -
  • Linear Regression and the SVD
    • -
  • What does it mean?
    • -
  • And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
    • -
  • Code for SVD and Inversion of Matrices
    • -
  • Inverse of Rectangular Matrix
    • -
  • Ridge and LASSO Regression
    • -
  • From OLS to Ridge and Lasso
    • -
  • Deriving the Ridge Regression Equations
    • -
  • Note on Scikit-Learn
    • -
  • Comparison with OLS
    • -
  • SVD analysis
    • -
  • Interpreting the Ridge results
    • -
  • More interpretations
    • -
  • Deriving the Lasso Regression Equations
    • -
  • Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
    • -
  • Ridge Regression
    • -
  • Lasso Regression
    • -
  • Yet another Example
    • -
  • The OLS case
    • -
  • The Ridge case
    • -
  • Writing the Cost Function
    • -
  • Lasso case
    • -
  • The first Case
    • -
  • Simple code for solving the above problem
    • -
  • With Lasso Regression
    • -
  • Another Example, now with a polynomial fit
    • -
  • To think about, first part
    • -
  • More thinking
    • -
  • Still thinking
    • -
  • What does centering (subtracting the mean values) mean mathematically?
    • -
  • Further Manipulations
    • -
  • Wrapping it up
    • -
  • Linear Regression code, Intercept handling first
    • -
  • Code Examples
    • -
  • Taking out the mean
    • -
  • Friday September 9
    • -
  • Linking the regression analysis with a statistical interpretation
    • -
  • Assumptions made
    • -
  • Expectation value and variance
    • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
    • -
  • Deriving OLS from a probability distribution
    • -
  • Independent and Identically Distrubuted (iid)
    • -
  • Maximum Likelihood Estimation (MLE)
    • -
  • A new Cost Function
    • -
  • More basic Statistics and Bayes' theorem
    • -
  • Marginal Probability
    • -
  • Conditional Probability
    • -
  • Bayes' Theorem
    • -
  • Interpretations of Bayes' Theorem
    • -
  • Example of Usage of Bayes' theorem
    • -
  • Doing it correctly
    • -
  • Bayes' Theorem and Ridge and Lasso Regression
    • -
  • Test Function for what happens with OLS, Ridge and Lasso
    • -
  • Invoking Bayes' theorem
    • -
  • Ridge and Bayes
    • -
  • Lasso and Bayes
    • -
  • Exercise 1: mean values and variances in linear regression
    • -
  • Exercise 2: Adding Ridge and Lasso Regression
    • +
  • Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
    • +
  • Linear Regression and the SVD
    • +
  • What does it mean?
    • +
  • And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
    • +
  • Code for SVD and Inversion of Matrices
    • +
  • Inverse of Rectangular Matrix
    • +
  • Ridge and LASSO Regression
    • +
  • From OLS to Ridge and Lasso
    • +
  • Deriving the Ridge Regression Equations
    • +
  • Note on Scikit-Learn
    • +
  • Comparison with OLS
    • +
  • SVD analysis
    • +
  • Interpreting the Ridge results
    • +
  • More interpretations
    • +
  • Deriving the Lasso Regression Equations
    • +
  • Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
    • +
  • Ridge Regression
    • +
  • Lasso Regression
    • +
  • Yet another Example
    • +
  • The OLS case
    • +
  • The Ridge case
    • +
  • Writing the Cost Function
    • +
  • Lasso case
    • +
  • The first Case
    • +
  • Simple code for solving the above problem
    • +
  • With Lasso Regression
    • +
  • Another Example, now with a polynomial fit
    • +
  • To think about, first part
    • +
  • More thinking
    • +
  • Still thinking
    • +
  • What does centering (subtracting the mean values) mean mathematically?
    • +
  • Further Manipulations
    • +
  • Wrapping it up
    • +
  • Linear Regression code, Intercept handling first
    • +
  • Code Examples
    • +
  • Taking out the mean
    • +
  • Friday September 9
    • +
  • Linking the regression analysis with a statistical interpretation
    • +
  • Assumptions made
    • +
  • Expectation value and variance
    • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
    • +
  • Deriving OLS from a probability distribution
    • +
  • Independent and Identically Distrubuted (iid)
    • +
  • Maximum Likelihood Estimation (MLE)
    • +
  • A new Cost Function
    • +
  • More basic Statistics and Bayes' theorem
    • +
  • Marginal Probability
    • +
  • Conditional Probability
    • +
  • Bayes' Theorem
    • +
  • Interpretations of Bayes' Theorem
    • +
  • Example of Usage of Bayes' theorem
    • +
  • Doing it correctly
    • +
  • Bayes' Theorem and Ridge and Lasso Regression
    • +
  • Test Function for what happens with OLS, Ridge and Lasso
    • +
  • Invoking Bayes' theorem
    • +
  • Ridge and Bayes
    • +
  • Lasso and Bayes
    • +
  • Exercise 1: mean values and variances in linear regression
    • +
  • Exercise 2: Adding Ridge and Lasso Regression
    • @@ -300,14 +298,8 @@ MathJax.Hub.Config({

    Plans for week 36

    Recommended Reading:

      @@ -331,7 +323,7 @@ MathJax.Hub.Config({
    1. 10
    2. 11
    3. ...
      4. -
    4. 60
      5. +
    5. 59
    6. »
      7. diff --git a/doc/pub/week36/html/._week36-bs002.html b/doc/pub/week36/html/._week36-bs002.html index ff32ce789..9e13c333e 100644 --- a/doc/pub/week36/html/._week36-bs002.html +++ b/doc/pub/week36/html/._week36-bs002.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    7. Plans for week 36
      8. -
    8. Thursday September 8
      9. -
    9. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      10. -
    10. Linear Regression and the SVD
      11. -
    11. What does it mean?
      12. -
    12. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      13. -
    13. Code for SVD and Inversion of Matrices
      14. -
    14. Inverse of Rectangular Matrix
      15. -
    15. Ridge and LASSO Regression
      16. -
    16. From OLS to Ridge and Lasso
      17. -
    17. Deriving the Ridge Regression Equations
      18. -
    18. Note on Scikit-Learn
      19. -
    19. Comparison with OLS
      20. -
    20. SVD analysis
      21. -
    21. Interpreting the Ridge results
      22. -
    22. More interpretations
      23. -
    23. Deriving the Lasso Regression Equations
      24. -
    24. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      25. -
    25. Ridge Regression
      26. -
    26. Lasso Regression
      27. -
    27. Yet another Example
      28. -
    28. The OLS case
      29. -
    29. The Ridge case
      30. -
    30. Writing the Cost Function
      31. -
    31. Lasso case
      32. -
    32. The first Case
      33. -
    33. Simple code for solving the above problem
      34. -
    34. With Lasso Regression
      35. -
    35. Another Example, now with a polynomial fit
      36. -
    36. To think about, first part
      37. -
    37. More thinking
      38. -
    38. Still thinking
      39. -
    39. What does centering (subtracting the mean values) mean mathematically?
      40. -
    40. Further Manipulations
      41. -
    41. Wrapping it up
      42. -
    42. Linear Regression code, Intercept handling first
      43. -
    43. Code Examples
      44. -
    44. Taking out the mean
      45. -
    45. Friday September 9
      46. -
    46. Linking the regression analysis with a statistical interpretation
      47. -
    47. Assumptions made
      48. -
    48. Expectation value and variance
      49. -
    49. Expectation value and variance for \( \boldsymbol{\beta} \)
      50. -
    50. Deriving OLS from a probability distribution
      51. -
    51. Independent and Identically Distrubuted (iid)
      52. -
    52. Maximum Likelihood Estimation (MLE)
      53. -
    53. A new Cost Function
      54. -
    54. More basic Statistics and Bayes' theorem
      55. -
    55. Marginal Probability
      56. -
    56. Conditional Probability
      57. -
    57. Bayes' Theorem
      58. -
    58. Interpretations of Bayes' Theorem
      59. -
    59. Example of Usage of Bayes' theorem
      60. -
    60. Doing it correctly
      61. -
    61. Bayes' Theorem and Ridge and Lasso Regression
      62. -
    62. Test Function for what happens with OLS, Ridge and Lasso
      63. -
    63. Invoking Bayes' theorem
      64. -
    64. Ridge and Bayes
      65. -
    65. Lasso and Bayes
      66. -
    66. Exercise 1: mean values and variances in linear regression
      67. -
    67. Exercise 2: Adding Ridge and Lasso Regression
      68. +
    68. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      69. +
    69. Linear Regression and the SVD
      70. +
    70. What does it mean?
      71. +
    71. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      72. +
    72. Code for SVD and Inversion of Matrices
      73. +
    73. Inverse of Rectangular Matrix
      74. +
    74. Ridge and LASSO Regression
      75. +
    75. From OLS to Ridge and Lasso
      76. +
    76. Deriving the Ridge Regression Equations
      77. +
    77. Note on Scikit-Learn
      78. +
    78. Comparison with OLS
      79. +
    79. SVD analysis
      80. +
    80. Interpreting the Ridge results
      81. +
    81. More interpretations
      82. +
    82. Deriving the Lasso Regression Equations
      83. +
    83. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      84. +
    84. Ridge Regression
      85. +
    85. Lasso Regression
      86. +
    86. Yet another Example
      87. +
    87. The OLS case
      88. +
    88. The Ridge case
      89. +
    89. Writing the Cost Function
      90. +
    90. Lasso case
      91. +
    91. The first Case
      92. +
    92. Simple code for solving the above problem
      93. +
    93. With Lasso Regression
      94. +
    94. Another Example, now with a polynomial fit
      95. +
    95. To think about, first part
      96. +
    96. More thinking
      97. +
    97. Still thinking
      98. +
    98. What does centering (subtracting the mean values) mean mathematically?
      99. +
    99. Further Manipulations
      100. +
    100. Wrapping it up
      101. +
    101. Linear Regression code, Intercept handling first
      102. +
    102. Code Examples
      103. +
    103. Taking out the mean
      104. +
    104. Friday September 9
      105. +
    105. Linking the regression analysis with a statistical interpretation
      106. +
    106. Assumptions made
      107. +
    107. Expectation value and variance
      108. +
    108. Expectation value and variance for \( \boldsymbol{\beta} \)
      109. +
    109. Deriving OLS from a probability distribution
      110. +
    110. Independent and Identically Distrubuted (iid)
      111. +
    111. Maximum Likelihood Estimation (MLE)
      112. +
    112. A new Cost Function
      113. +
    113. More basic Statistics and Bayes' theorem
      114. +
    114. Marginal Probability
      115. +
    115. Conditional Probability
      116. +
    116. Bayes' Theorem
      117. +
    117. Interpretations of Bayes' Theorem
      118. +
    118. Example of Usage of Bayes' theorem
      119. +
    119. Doing it correctly
      120. +
    120. Bayes' Theorem and Ridge and Lasso Regression
      121. +
    121. Test Function for what happens with OLS, Ridge and Lasso
      122. +
    122. Invoking Bayes' theorem
      123. +
    123. Ridge and Bayes
      124. +
    124. Lasso and Bayes
      125. +
    125. Exercise 1: mean values and variances in linear regression
      126. +
    126. Exercise 2: Adding Ridge and Lasso Regression
      127. @@ -297,7 +295,7 @@ MathJax.Hub.Config({

       

       

       

      -

      Thursday September 8

      +

      Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples

      @@ -316,7 +314,7 @@ MathJax.Hub.Config({

    127. 11
    128. 12
    129. ...
      130. -
    130. 60
      131. +
    131. 59
    132. »
      133. diff --git a/doc/pub/week36/html/._week36-bs003.html b/doc/pub/week36/html/._week36-bs003.html index 016405d5f..ff7746d93 100644 --- a/doc/pub/week36/html/._week36-bs003.html +++ b/doc/pub/week36/html/._week36-bs003.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    133. Plans for week 36
      134. -
    134. Thursday September 8
      135. -
    135. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      136. -
    136. Linear Regression and the SVD
      137. -
    137. What does it mean?
      138. -
    138. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      139. -
    139. Code for SVD and Inversion of Matrices
      140. -
    140. Inverse of Rectangular Matrix
      141. -
    141. Ridge and LASSO Regression
      142. -
    142. From OLS to Ridge and Lasso
      143. -
    143. Deriving the Ridge Regression Equations
      144. -
    144. Note on Scikit-Learn
      145. -
    145. Comparison with OLS
      146. -
    146. SVD analysis
      147. -
    147. Interpreting the Ridge results
      148. -
    148. More interpretations
      149. -
    149. Deriving the Lasso Regression Equations
      150. -
    150. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      151. -
    151. Ridge Regression
      152. -
    152. Lasso Regression
      153. -
    153. Yet another Example
      154. -
    154. The OLS case
      155. -
    155. The Ridge case
      156. -
    156. Writing the Cost Function
      157. -
    157. Lasso case
      158. -
    158. The first Case
      159. -
    159. Simple code for solving the above problem
      160. -
    160. With Lasso Regression
      161. -
    161. Another Example, now with a polynomial fit
      162. -
    162. To think about, first part
      163. -
    163. More thinking
      164. -
    164. Still thinking
      165. -
    165. What does centering (subtracting the mean values) mean mathematically?
      166. -
    166. Further Manipulations
      167. -
    167. Wrapping it up
      168. -
    168. Linear Regression code, Intercept handling first
      169. -
    169. Code Examples
      170. -
    170. Taking out the mean
      171. -
    171. Friday September 9
      172. -
    172. Linking the regression analysis with a statistical interpretation
      173. -
    173. Assumptions made
      174. -
    174. Expectation value and variance
      175. -
    175. Expectation value and variance for \( \boldsymbol{\beta} \)
      176. -
    176. Deriving OLS from a probability distribution
      177. -
    177. Independent and Identically Distrubuted (iid)
      178. -
    178. Maximum Likelihood Estimation (MLE)
      179. -
    179. A new Cost Function
      180. -
    180. More basic Statistics and Bayes' theorem
      181. -
    181. Marginal Probability
      182. -
    182. Conditional Probability
      183. -
    183. Bayes' Theorem
      184. -
    184. Interpretations of Bayes' Theorem
      185. -
    185. Example of Usage of Bayes' theorem
      186. -
    186. Doing it correctly
      187. -
    187. Bayes' Theorem and Ridge and Lasso Regression
      188. -
    188. Test Function for what happens with OLS, Ridge and Lasso
      189. -
    189. Invoking Bayes' theorem
      190. -
    190. Ridge and Bayes
      191. -
    191. Lasso and Bayes
      192. -
    192. Exercise 1: mean values and variances in linear regression
      193. -
    193. Exercise 2: Adding Ridge and Lasso Regression
      194. +
    194. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      195. +
    195. Linear Regression and the SVD
      196. +
    196. What does it mean?
      197. +
    197. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      198. +
    198. Code for SVD and Inversion of Matrices
      199. +
    199. Inverse of Rectangular Matrix
      200. +
    200. Ridge and LASSO Regression
      201. +
    201. From OLS to Ridge and Lasso
      202. +
    202. Deriving the Ridge Regression Equations
      203. +
    203. Note on Scikit-Learn
      204. +
    204. Comparison with OLS
      205. +
    205. SVD analysis
      206. +
    206. Interpreting the Ridge results
      207. +
    207. More interpretations
      208. +
    208. Deriving the Lasso Regression Equations
      209. +
    209. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      210. +
    210. Ridge Regression
      211. +
    211. Lasso Regression
      212. +
    212. Yet another Example
      213. +
    213. The OLS case
      214. +
    214. The Ridge case
      215. +
    215. Writing the Cost Function
      216. +
    216. Lasso case
      217. +
    217. The first Case
      218. +
    218. Simple code for solving the above problem
      219. +
    219. With Lasso Regression
      220. +
    220. Another Example, now with a polynomial fit
      221. +
    221. To think about, first part
      222. +
    222. More thinking
      223. +
    223. Still thinking
      224. +
    224. What does centering (subtracting the mean values) mean mathematically?
      225. +
    225. Further Manipulations
      226. +
    226. Wrapping it up
      227. +
    227. Linear Regression code, Intercept handling first
      228. +
    228. Code Examples
      229. +
    229. Taking out the mean
      230. +
    230. Friday September 9
      231. +
    231. Linking the regression analysis with a statistical interpretation
      232. +
    232. Assumptions made
      233. +
    233. Expectation value and variance
      234. +
    234. Expectation value and variance for \( \boldsymbol{\beta} \)
      235. +
    235. Deriving OLS from a probability distribution
      236. +
    236. Independent and Identically Distrubuted (iid)
      237. +
    237. Maximum Likelihood Estimation (MLE)
      238. +
    238. A new Cost Function
      239. +
    239. More basic Statistics and Bayes' theorem
      240. +
    240. Marginal Probability
      241. +
    241. Conditional Probability
      242. +
    242. Bayes' Theorem
      243. +
    243. Interpretations of Bayes' Theorem
      244. +
    244. Example of Usage of Bayes' theorem
      245. +
    245. Doing it correctly
      246. +
    246. Bayes' Theorem and Ridge and Lasso Regression
      247. +
    247. Test Function for what happens with OLS, Ridge and Lasso
      248. +
    248. Invoking Bayes' theorem
      249. +
    249. Ridge and Bayes
      250. +
    250. Lasso and Bayes
      251. +
    251. Exercise 1: mean values and variances in linear regression
      252. +
    252. Exercise 2: Adding Ridge and Lasso Regression
      253. @@ -297,7 +295,39 @@ MathJax.Hub.Config({

       

       

       

      -

      Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples

      +

      Linear Regression and the SVD

      + +

      We used the SVD to analyse the matrix to invert in ordinary lineat regression

      +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +

      Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix

      +$$ +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\end{bmatrix}, +$$ + +

      where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is

      + +$$ +\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ + 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ + 0 & 0 & \sigma_2 & \dots & 0 & 0 \\ + 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ + 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ +\end{bmatrix}, +$$ + +

      meaning we can write

      +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. +$$ + +

      Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

      +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. +$$ +

      @@ -317,7 +347,7 @@ MathJax.Hub.Config({

    253. 12
    254. 13
    255. ...
      256. -
    256. 60
      257. +
    257. 59
    258. »
      259. diff --git a/doc/pub/week36/html/._week36-bs004.html b/doc/pub/week36/html/._week36-bs004.html index 0bef54934..9ec432f70 100644 --- a/doc/pub/week36/html/._week36-bs004.html +++ b/doc/pub/week36/html/._week36-bs004.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    259. Plans for week 36
      260. -
    260. Thursday September 8
      261. -
    261. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      262. -
    262. Linear Regression and the SVD
      263. -
    263. What does it mean?
      264. -
    264. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      265. -
    265. Code for SVD and Inversion of Matrices
      266. -
    266. Inverse of Rectangular Matrix
      267. -
    267. Ridge and LASSO Regression
      268. -
    268. From OLS to Ridge and Lasso
      269. -
    269. Deriving the Ridge Regression Equations
      270. -
    270. Note on Scikit-Learn
      271. -
    271. Comparison with OLS
      272. -
    272. SVD analysis
      273. -
    273. Interpreting the Ridge results
      274. -
    274. More interpretations
      275. -
    275. Deriving the Lasso Regression Equations
      276. -
    276. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      277. -
    277. Ridge Regression
      278. -
    278. Lasso Regression
      279. -
    279. Yet another Example
      280. -
    280. The OLS case
      281. -
    281. The Ridge case
      282. -
    282. Writing the Cost Function
      283. -
    283. Lasso case
      284. -
    284. The first Case
      285. -
    285. Simple code for solving the above problem
      286. -
    286. With Lasso Regression
      287. -
    287. Another Example, now with a polynomial fit
      288. -
    288. To think about, first part
      289. -
    289. More thinking
      290. -
    290. Still thinking
      291. -
    291. What does centering (subtracting the mean values) mean mathematically?
      292. -
    292. Further Manipulations
      293. -
    293. Wrapping it up
      294. -
    294. Linear Regression code, Intercept handling first
      295. -
    295. Code Examples
      296. -
    296. Taking out the mean
      297. -
    297. Friday September 9
      298. -
    298. Linking the regression analysis with a statistical interpretation
      299. -
    299. Assumptions made
      300. -
    300. Expectation value and variance
      301. -
    301. Expectation value and variance for \( \boldsymbol{\beta} \)
      302. -
    302. Deriving OLS from a probability distribution
      303. -
    303. Independent and Identically Distrubuted (iid)
      304. -
    304. Maximum Likelihood Estimation (MLE)
      305. -
    305. A new Cost Function
      306. -
    306. More basic Statistics and Bayes' theorem
      307. -
    307. Marginal Probability
      308. -
    308. Conditional Probability
      309. -
    309. Bayes' Theorem
      310. -
    310. Interpretations of Bayes' Theorem
      311. -
    311. Example of Usage of Bayes' theorem
      312. -
    312. Doing it correctly
      313. -
    313. Bayes' Theorem and Ridge and Lasso Regression
      314. -
    314. Test Function for what happens with OLS, Ridge and Lasso
      315. -
    315. Invoking Bayes' theorem
      316. -
    316. Ridge and Bayes
      317. -
    317. Lasso and Bayes
      318. -
    318. Exercise 1: mean values and variances in linear regression
      319. -
    319. Exercise 2: Adding Ridge and Lasso Regression
      320. +
    320. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      321. +
    321. Linear Regression and the SVD
      322. +
    322. What does it mean?
      323. +
    323. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      324. +
    324. Code for SVD and Inversion of Matrices
      325. +
    325. Inverse of Rectangular Matrix
      326. +
    326. Ridge and LASSO Regression
      327. +
    327. From OLS to Ridge and Lasso
      328. +
    328. Deriving the Ridge Regression Equations
      329. +
    329. Note on Scikit-Learn
      330. +
    330. Comparison with OLS
      331. +
    331. SVD analysis
      332. +
    332. Interpreting the Ridge results
      333. +
    333. More interpretations
      334. +
    334. Deriving the Lasso Regression Equations
      335. +
    335. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      336. +
    336. Ridge Regression
      337. +
    337. Lasso Regression
      338. +
    338. Yet another Example
      339. +
    339. The OLS case
      340. +
    340. The Ridge case
      341. +
    341. Writing the Cost Function
      342. +
    342. Lasso case
      343. +
    343. The first Case
      344. +
    344. Simple code for solving the above problem
      345. +
    345. With Lasso Regression
      346. +
    346. Another Example, now with a polynomial fit
      347. +
    347. To think about, first part
      348. +
    348. More thinking
      349. +
    349. Still thinking
      350. +
    350. What does centering (subtracting the mean values) mean mathematically?
      351. +
    351. Further Manipulations
      352. +
    352. Wrapping it up
      353. +
    353. Linear Regression code, Intercept handling first
      354. +
    354. Code Examples
      355. +
    355. Taking out the mean
      356. +
    356. Friday September 9
      357. +
    357. Linking the regression analysis with a statistical interpretation
      358. +
    358. Assumptions made
      359. +
    359. Expectation value and variance
      360. +
    360. Expectation value and variance for \( \boldsymbol{\beta} \)
      361. +
    361. Deriving OLS from a probability distribution
      362. +
    362. Independent and Identically Distrubuted (iid)
      363. +
    363. Maximum Likelihood Estimation (MLE)
      364. +
    364. A new Cost Function
      365. +
    365. More basic Statistics and Bayes' theorem
      366. +
    366. Marginal Probability
      367. +
    367. Conditional Probability
      368. +
    368. Bayes' Theorem
      369. +
    369. Interpretations of Bayes' Theorem
      370. +
    370. Example of Usage of Bayes' theorem
      371. +
    371. Doing it correctly
      372. +
    372. Bayes' Theorem and Ridge and Lasso Regression
      373. +
    373. Test Function for what happens with OLS, Ridge and Lasso
      374. +
    374. Invoking Bayes' theorem
      375. +
    375. Ridge and Bayes
      376. +
    376. Lasso and Bayes
      377. +
    377. Exercise 1: mean values and variances in linear regression
      378. +
    378. Exercise 2: Adding Ridge and Lasso Regression
      379. @@ -297,39 +295,45 @@ MathJax.Hub.Config({

       

       

       

      -

      Linear Regression and the SVD

      +

      What does it mean?

      -

      We used the SVD to analyse the matrix to invert in ordinary lineat regression

      -$$ -\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. -$$ - -

      Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix

      -$$ -\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\end{bmatrix}, -$$ - -

      where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is

      +

      This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues +given by the singular values squared, that is +

      $$ -\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ - 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ - 0 & 0 & \sigma_2 & \dots & 0 & 0 \\ - 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ - 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ -\end{bmatrix}, +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

      meaning we can write

      +

      In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that +the columns of \( \boldsymbol{V} \) are the eigenvectors of +\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of +\( \boldsymbol{X} \) in a descending order, it means that the column vectors +\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they +encode from the columns of \( \boldsymbol{X} \). +

      + +

      Note that these are also the eigenvectors and eigenvalues of the +Hessian matrix. +

      + +

      If we now recall the definition of the covariance matrix (not using +Bessel's correction) we have +

      + $$ -\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. -$$ - -

      Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

      -$$ -\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. +\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, $$ +

      meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +the number of samples) are the eigenvalues of the covariance +matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square +root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is +self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the +absolute value of the eigenvalues of \( \boldsymbol{X} \). +

      @@ -350,7 +354,7 @@ $$

    379. 13
    380. 14
    381. ...
      382. -
    382. 60
      383. +
    383. 59
    384. »
      385. diff --git a/doc/pub/week36/html/._week36-bs005.html b/doc/pub/week36/html/._week36-bs005.html index e468cd6b6..395ad8925 100644 --- a/doc/pub/week36/html/._week36-bs005.html +++ b/doc/pub/week36/html/._week36-bs005.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    385. Plans for week 36
      386. -
    386. Thursday September 8
      387. -
    387. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      388. -
    388. Linear Regression and the SVD
      389. -
    389. What does it mean?
      390. -
    390. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      391. -
    391. Code for SVD and Inversion of Matrices
      392. -
    392. Inverse of Rectangular Matrix
      393. -
    393. Ridge and LASSO Regression
      394. -
    394. From OLS to Ridge and Lasso
      395. -
    395. Deriving the Ridge Regression Equations
      396. -
    396. Note on Scikit-Learn
      397. -
    397. Comparison with OLS
      398. -
    398. SVD analysis
      399. -
    399. Interpreting the Ridge results
      400. -
    400. More interpretations
      401. -
    401. Deriving the Lasso Regression Equations
      402. -
    402. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      403. -
    403. Ridge Regression
      404. -
    404. Lasso Regression
      405. -
    405. Yet another Example
      406. -
    406. The OLS case
      407. -
    407. The Ridge case
      408. -
    408. Writing the Cost Function
      409. -
    409. Lasso case
      410. -
    410. The first Case
      411. -
    411. Simple code for solving the above problem
      412. -
    412. With Lasso Regression
      413. -
    413. Another Example, now with a polynomial fit
      414. -
    414. To think about, first part
      415. -
    415. More thinking
      416. -
    416. Still thinking
      417. -
    417. What does centering (subtracting the mean values) mean mathematically?
      418. -
    418. Further Manipulations
      419. -
    419. Wrapping it up
      420. -
    420. Linear Regression code, Intercept handling first
      421. -
    421. Code Examples
      422. -
    422. Taking out the mean
      423. -
    423. Friday September 9
      424. -
    424. Linking the regression analysis with a statistical interpretation
      425. -
    425. Assumptions made
      426. -
    426. Expectation value and variance
      427. -
    427. Expectation value and variance for \( \boldsymbol{\beta} \)
      428. -
    428. Deriving OLS from a probability distribution
      429. -
    429. Independent and Identically Distrubuted (iid)
      430. -
    430. Maximum Likelihood Estimation (MLE)
      431. -
    431. A new Cost Function
      432. -
    432. More basic Statistics and Bayes' theorem
      433. -
    433. Marginal Probability
      434. -
    434. Conditional Probability
      435. -
    435. Bayes' Theorem
      436. -
    436. Interpretations of Bayes' Theorem
      437. -
    437. Example of Usage of Bayes' theorem
      438. -
    438. Doing it correctly
      439. -
    439. Bayes' Theorem and Ridge and Lasso Regression
      440. -
    440. Test Function for what happens with OLS, Ridge and Lasso
      441. -
    441. Invoking Bayes' theorem
      442. -
    442. Ridge and Bayes
      443. -
    443. Lasso and Bayes
      444. -
    444. Exercise 1: mean values and variances in linear regression
      445. -
    445. Exercise 2: Adding Ridge and Lasso Regression
      446. +
    446. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      447. +
    447. Linear Regression and the SVD
      448. +
    448. What does it mean?
      449. +
    449. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      450. +
    450. Code for SVD and Inversion of Matrices
      451. +
    451. Inverse of Rectangular Matrix
      452. +
    452. Ridge and LASSO Regression
      453. +
    453. From OLS to Ridge and Lasso
      454. +
    454. Deriving the Ridge Regression Equations
      455. +
    455. Note on Scikit-Learn
      456. +
    456. Comparison with OLS
      457. +
    457. SVD analysis
      458. +
    458. Interpreting the Ridge results
      459. +
    459. More interpretations
      460. +
    460. Deriving the Lasso Regression Equations
      461. +
    461. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      462. +
    462. Ridge Regression
      463. +
    463. Lasso Regression
      464. +
    464. Yet another Example
      465. +
    465. The OLS case
      466. +
    466. The Ridge case
      467. +
    467. Writing the Cost Function
      468. +
    468. Lasso case
      469. +
    469. The first Case
      470. +
    470. Simple code for solving the above problem
      471. +
    471. With Lasso Regression
      472. +
    472. Another Example, now with a polynomial fit
      473. +
    473. To think about, first part
      474. +
    474. More thinking
      475. +
    475. Still thinking
      476. +
    476. What does centering (subtracting the mean values) mean mathematically?
      477. +
    477. Further Manipulations
      478. +
    478. Wrapping it up
      479. +
    479. Linear Regression code, Intercept handling first
      480. +
    480. Code Examples
      481. +
    481. Taking out the mean
      482. +
    482. Friday September 9
      483. +
    483. Linking the regression analysis with a statistical interpretation
      484. +
    484. Assumptions made
      485. +
    485. Expectation value and variance
      486. +
    486. Expectation value and variance for \( \boldsymbol{\beta} \)
      487. +
    487. Deriving OLS from a probability distribution
      488. +
    488. Independent and Identically Distrubuted (iid)
      489. +
    489. Maximum Likelihood Estimation (MLE)
      490. +
    490. A new Cost Function
      491. +
    491. More basic Statistics and Bayes' theorem
      492. +
    492. Marginal Probability
      493. +
    493. Conditional Probability
      494. +
    494. Bayes' Theorem
      495. +
    495. Interpretations of Bayes' Theorem
      496. +
    496. Example of Usage of Bayes' theorem
      497. +
    497. Doing it correctly
      498. +
    498. Bayes' Theorem and Ridge and Lasso Regression
      499. +
    499. Test Function for what happens with OLS, Ridge and Lasso
      500. +
    500. Invoking Bayes' theorem
      501. +
    501. Ridge and Bayes
      502. +
    502. Lasso and Bayes
      503. +
    503. Exercise 1: mean values and variances in linear regression
      504. +
    504. Exercise 2: Adding Ridge and Lasso Regression
      505. @@ -297,44 +295,38 @@ MathJax.Hub.Config({

       

       

       

      -

      What does it mean?

      +

      And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

      -

      This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) -are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues -given by the singular values squared, that is -

      +

      For \( \boldsymbol{X}\boldsymbol{X}^T \) we found

      $$ -\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ -

      In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive -square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that -the columns of \( \boldsymbol{V} \) are the eigenvectors of -\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of -\( \boldsymbol{X} \) in a descending order, it means that the column vectors -\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they -encode from the columns of \( \boldsymbol{X} \). -

      - -

      Note that these are also the eigenvectors and eigenvalues of the -Hessian matrix. -

      - -

      If we now recall the definition of the covariance matrix (not using -Bessel's correction) we have -

      - +

      Since the matrices here have dimension \( n\times n \), we have

      $$ -\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ -

      meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( -the number of samples) are the eigenvalues of the covariance -matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square -root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is -self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the -absolute value of the eigenvalues of \( \boldsymbol{X} \). +

      leading to

      +$$ +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. +$$ + +

      Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem

      +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. +$$ + +

      It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +the non-zero singular values plus now a series of zeros. The column +vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and +measure how much correlations are contained in the rows of \( \boldsymbol{X} \). +

      + +

      Since we will mainly be interested in the correlations among the features +of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular +values and the column vectors of \( \boldsymbol{V} \).

      @@ -357,7 +349,7 @@ absolute value of the eigenvalues of \( \boldsymbol{X} \).

    505. 14
    506. 15
    507. ...
      508. -
    508. 60
      509. +
    509. 59
    510. »
      511. diff --git a/doc/pub/week36/html/._week36-bs006.html b/doc/pub/week36/html/._week36-bs006.html index 395a166a7..996408460 100644 --- a/doc/pub/week36/html/._week36-bs006.html +++ b/doc/pub/week36/html/._week36-bs006.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    511. Plans for week 36
      512. -
    512. Thursday September 8
      513. -
    513. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      514. -
    514. Linear Regression and the SVD
      515. -
    515. What does it mean?
      516. -
    516. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      517. -
    517. Code for SVD and Inversion of Matrices
      518. -
    518. Inverse of Rectangular Matrix
      519. -
    519. Ridge and LASSO Regression
      520. -
    520. From OLS to Ridge and Lasso
      521. -
    521. Deriving the Ridge Regression Equations
      522. -
    522. Note on Scikit-Learn
      523. -
    523. Comparison with OLS
      524. -
    524. SVD analysis
      525. -
    525. Interpreting the Ridge results
      526. -
    526. More interpretations
      527. -
    527. Deriving the Lasso Regression Equations
      528. -
    528. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      529. -
    529. Ridge Regression
      530. -
    530. Lasso Regression
      531. -
    531. Yet another Example
      532. -
    532. The OLS case
      533. -
    533. The Ridge case
      534. -
    534. Writing the Cost Function
      535. -
    535. Lasso case
      536. -
    536. The first Case
      537. -
    537. Simple code for solving the above problem
      538. -
    538. With Lasso Regression
      539. -
    539. Another Example, now with a polynomial fit
      540. -
    540. To think about, first part
      541. -
    541. More thinking
      542. -
    542. Still thinking
      543. -
    543. What does centering (subtracting the mean values) mean mathematically?
      544. -
    544. Further Manipulations
      545. -
    545. Wrapping it up
      546. -
    546. Linear Regression code, Intercept handling first
      547. -
    547. Code Examples
      548. -
    548. Taking out the mean
      549. -
    549. Friday September 9
      550. -
    550. Linking the regression analysis with a statistical interpretation
      551. -
    551. Assumptions made
      552. -
    552. Expectation value and variance
      553. -
    553. Expectation value and variance for \( \boldsymbol{\beta} \)
      554. -
    554. Deriving OLS from a probability distribution
      555. -
    555. Independent and Identically Distrubuted (iid)
      556. -
    556. Maximum Likelihood Estimation (MLE)
      557. -
    557. A new Cost Function
      558. -
    558. More basic Statistics and Bayes' theorem
      559. -
    559. Marginal Probability
      560. -
    560. Conditional Probability
      561. -
    561. Bayes' Theorem
      562. -
    562. Interpretations of Bayes' Theorem
      563. -
    563. Example of Usage of Bayes' theorem
      564. -
    564. Doing it correctly
      565. -
    565. Bayes' Theorem and Ridge and Lasso Regression
      566. -
    566. Test Function for what happens with OLS, Ridge and Lasso
      567. -
    567. Invoking Bayes' theorem
      568. -
    568. Ridge and Bayes
      569. -
    569. Lasso and Bayes
      570. -
    570. Exercise 1: mean values and variances in linear regression
      571. -
    571. Exercise 2: Adding Ridge and Lasso Regression
      572. +
    572. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      573. +
    573. Linear Regression and the SVD
      574. +
    574. What does it mean?
      575. +
    575. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      576. +
    576. Code for SVD and Inversion of Matrices
      577. +
    577. Inverse of Rectangular Matrix
      578. +
    578. Ridge and LASSO Regression
      579. +
    579. From OLS to Ridge and Lasso
      580. +
    580. Deriving the Ridge Regression Equations
      581. +
    581. Note on Scikit-Learn
      582. +
    582. Comparison with OLS
      583. +
    583. SVD analysis
      584. +
    584. Interpreting the Ridge results
      585. +
    585. More interpretations
      586. +
    586. Deriving the Lasso Regression Equations
      587. +
    587. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      588. +
    588. Ridge Regression
      589. +
    589. Lasso Regression
      590. +
    590. Yet another Example
      591. +
    591. The OLS case
      592. +
    592. The Ridge case
      593. +
    593. Writing the Cost Function
      594. +
    594. Lasso case
      595. +
    595. The first Case
      596. +
    596. Simple code for solving the above problem
      597. +
    597. With Lasso Regression
      598. +
    598. Another Example, now with a polynomial fit
      599. +
    599. To think about, first part
      600. +
    600. More thinking
      601. +
    601. Still thinking
      602. +
    602. What does centering (subtracting the mean values) mean mathematically?
      603. +
    603. Further Manipulations
      604. +
    604. Wrapping it up
      605. +
    605. Linear Regression code, Intercept handling first
      606. +
    606. Code Examples
      607. +
    607. Taking out the mean
      608. +
    608. Friday September 9
      609. +
    609. Linking the regression analysis with a statistical interpretation
      610. +
    610. Assumptions made
      611. +
    611. Expectation value and variance
      612. +
    612. Expectation value and variance for \( \boldsymbol{\beta} \)
      613. +
    613. Deriving OLS from a probability distribution
      614. +
    614. Independent and Identically Distrubuted (iid)
      615. +
    615. Maximum Likelihood Estimation (MLE)
      616. +
    616. A new Cost Function
      617. +
    617. More basic Statistics and Bayes' theorem
      618. +
    618. Marginal Probability
      619. +
    619. Conditional Probability
      620. +
    620. Bayes' Theorem
      621. +
    621. Interpretations of Bayes' Theorem
      622. +
    622. Example of Usage of Bayes' theorem
      623. +
    623. Doing it correctly
      624. +
    624. Bayes' Theorem and Ridge and Lasso Regression
      625. +
    625. Test Function for what happens with OLS, Ridge and Lasso
      626. +
    626. Invoking Bayes' theorem
      627. +
    627. Ridge and Bayes
      628. +
    628. Lasso and Bayes
      629. +
    629. Exercise 1: mean values and variances in linear regression
      630. +
    630. Exercise 2: Adding Ridge and Lasso Regression
      631. @@ -297,39 +295,87 @@ MathJax.Hub.Config({

       

       

       

      -

      And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

      +

      Code for SVD and Inversion of Matrices

      -

      For \( \boldsymbol{X}\boldsymbol{X}^T \) we found

      - -$$ -\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. -$$ - -

      Since the matrices here have dimension \( n\times n \), we have

      -$$ -\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, -$$ - -

      leading to

      -$$ -\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. -$$ - -

      Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem

      -$$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. -$$ - -

      It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by -the non-zero singular values plus now a series of zeros. The column -vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and -measure how much correlations are contained in the rows of \( \boldsymbol{X} \). +

      How do we use the SVD to invert a matrix \( \boldsymbol{X}^\boldsymbol{X} \) which is singular or near singular? +The simple answer is to use the linear algebra function for pseudoinvers, that is

      -

      Since we will mainly be interested in the correlations among the features -of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular -values and the column vectors of \( \boldsymbol{V} \). -

      + +
      +
      +
      +
      +
      + 
      Ainv = np.linlag.pinv(A)
+
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      + +

      Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD.

      + + + +
      +
      +
      +
      +
      + 
      import numpy as np
+# SVD inversion
+def SVDinv(A):
+    ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
+    SVD is numerically more stable than the inversion algorithms provided by
+    numpy and scipy.linalg at the cost of being slower.
+    '''
+    U, s, VT = np.linalg.svd(A)
+    print('test U')
+    print( (np.transpose(U) @ U - U @np.transpose(U)))
+    print('test VT')
+    print( (np.transpose(VT) @ VT - VT @np.transpose(VT)))
+
+
+    D = np.zeros((len(U),len(VT)))
+    D = np.diag(s)
+    UT = np.transpose(U); V = np.transpose(VT); invD = np.linalg.inv(D)
+    return np.matmul(V,np.matmul(invD,UT))
+
+
+#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])
+# Non-singular square matrix
+X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])
+print(X)
+A = np.transpose(X) @ X
+# Brute force inversion
+B = np.linalg.inv(A)  # here we could use np.linalg.pinv(A)
+C = SVDinv(A)
+print(np.abs(B-C))
+
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +

      @@ -352,7 +398,7 @@ values and the column vectors of \( \boldsymbol{V} \).

    631. 15
    632. 16
    633. ...
      634. -
    634. 60
      635. +
    635. 59
    636. »
      637. diff --git a/doc/pub/week36/html/._week36-bs007.html b/doc/pub/week36/html/._week36-bs007.html index 055ec9aae..3e6dd57ec 100644 --- a/doc/pub/week36/html/._week36-bs007.html +++ b/doc/pub/week36/html/._week36-bs007.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    637. Plans for week 36
      638. -
    638. Thursday September 8
      639. -
    639. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      640. -
    640. Linear Regression and the SVD
      641. -
    641. What does it mean?
      642. -
    642. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      643. -
    643. Code for SVD and Inversion of Matrices
      644. -
    644. Inverse of Rectangular Matrix
      645. -
    645. Ridge and LASSO Regression
      646. -
    646. From OLS to Ridge and Lasso
      647. -
    647. Deriving the Ridge Regression Equations
      648. -
    648. Note on Scikit-Learn
      649. -
    649. Comparison with OLS
      650. -
    650. SVD analysis
      651. -
    651. Interpreting the Ridge results
      652. -
    652. More interpretations
      653. -
    653. Deriving the Lasso Regression Equations
      654. -
    654. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      655. -
    655. Ridge Regression
      656. -
    656. Lasso Regression
      657. -
    657. Yet another Example
      658. -
    658. The OLS case
      659. -
    659. The Ridge case
      660. -
    660. Writing the Cost Function
      661. -
    661. Lasso case
      662. -
    662. The first Case
      663. -
    663. Simple code for solving the above problem
      664. -
    664. With Lasso Regression
      665. -
    665. Another Example, now with a polynomial fit
      666. -
    666. To think about, first part
      667. -
    667. More thinking
      668. -
    668. Still thinking
      669. -
    669. What does centering (subtracting the mean values) mean mathematically?
      670. -
    670. Further Manipulations
      671. -
    671. Wrapping it up
      672. -
    672. Linear Regression code, Intercept handling first
      673. -
    673. Code Examples
      674. -
    674. Taking out the mean
      675. -
    675. Friday September 9
      676. -
    676. Linking the regression analysis with a statistical interpretation
      677. -
    677. Assumptions made
      678. -
    678. Expectation value and variance
      679. -
    679. Expectation value and variance for \( \boldsymbol{\beta} \)
      680. -
    680. Deriving OLS from a probability distribution
      681. -
    681. Independent and Identically Distrubuted (iid)
      682. -
    682. Maximum Likelihood Estimation (MLE)
      683. -
    683. A new Cost Function
      684. -
    684. More basic Statistics and Bayes' theorem
      685. -
    685. Marginal Probability
      686. -
    686. Conditional Probability
      687. -
    687. Bayes' Theorem
      688. -
    688. Interpretations of Bayes' Theorem
      689. -
    689. Example of Usage of Bayes' theorem
      690. -
    690. Doing it correctly
      691. -
    691. Bayes' Theorem and Ridge and Lasso Regression
      692. -
    692. Test Function for what happens with OLS, Ridge and Lasso
      693. -
    693. Invoking Bayes' theorem
      694. -
    694. Ridge and Bayes
      695. -
    695. Lasso and Bayes
      696. -
    696. Exercise 1: mean values and variances in linear regression
      697. -
    697. Exercise 2: Adding Ridge and Lasso Regression
      698. +
    698. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      699. +
    699. Linear Regression and the SVD
      700. +
    700. What does it mean?
      701. +
    701. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      702. +
    702. Code for SVD and Inversion of Matrices
      703. +
    703. Inverse of Rectangular Matrix
      704. +
    704. Ridge and LASSO Regression
      705. +
    705. From OLS to Ridge and Lasso
      706. +
    706. Deriving the Ridge Regression Equations
      707. +
    707. Note on Scikit-Learn
      708. +
    708. Comparison with OLS
      709. +
    709. SVD analysis
      710. +
    710. Interpreting the Ridge results
      711. +
    711. More interpretations
      712. +
    712. Deriving the Lasso Regression Equations
      713. +
    713. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      714. +
    714. Ridge Regression
      715. +
    715. Lasso Regression
      716. +
    716. Yet another Example
      717. +
    717. The OLS case
      718. +
    718. The Ridge case
      719. +
    719. Writing the Cost Function
      720. +
    720. Lasso case
      721. +
    721. The first Case
      722. +
    722. Simple code for solving the above problem
      723. +
    723. With Lasso Regression
      724. +
    724. Another Example, now with a polynomial fit
      725. +
    725. To think about, first part
      726. +
    726. More thinking
      727. +
    727. Still thinking
      728. +
    728. What does centering (subtracting the mean values) mean mathematically?
      729. +
    729. Further Manipulations
      730. +
    730. Wrapping it up
      731. +
    731. Linear Regression code, Intercept handling first
      732. +
    732. Code Examples
      733. +
    733. Taking out the mean
      734. +
    734. Friday September 9
      735. +
    735. Linking the regression analysis with a statistical interpretation
      736. +
    736. Assumptions made
      737. +
    737. Expectation value and variance
      738. +
    738. Expectation value and variance for \( \boldsymbol{\beta} \)
      739. +
    739. Deriving OLS from a probability distribution
      740. +
    740. Independent and Identically Distrubuted (iid)
      741. +
    741. Maximum Likelihood Estimation (MLE)
      742. +
    742. A new Cost Function
      743. +
    743. More basic Statistics and Bayes' theorem
      744. +
    744. Marginal Probability
      745. +
    745. Conditional Probability
      746. +
    746. Bayes' Theorem
      747. +
    747. Interpretations of Bayes' Theorem
      748. +
    748. Example of Usage of Bayes' theorem
      749. +
    749. Doing it correctly
      750. +
    750. Bayes' Theorem and Ridge and Lasso Regression
      751. +
    751. Test Function for what happens with OLS, Ridge and Lasso
      752. +
    752. Invoking Bayes' theorem
      753. +
    753. Ridge and Bayes
      754. +
    754. Lasso and Bayes
      755. +
    755. Exercise 1: mean values and variances in linear regression
      756. +
    756. Exercise 2: Adding Ridge and Lasso Regression
      757. @@ -297,35 +295,24 @@ MathJax.Hub.Config({

       

       

       

      -

      Code for SVD and Inversion of Matrices

      +

      Inverse of Rectangular Matrix

      -

      How do we use the SVD to invert a matrix \( \boldsymbol{X}^\boldsymbol{X} \) which is singular or near singular? -The simple answer is to use the linear algebra function for pseudoinvers, that is +

      Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular.

      + +

      The pseudoinverse is the generalization of the matrix inverse for square matrices to +rectangular matrices where the number of rows and columns are not equal.

      - -
      -
      -
      -
      -
      - 
      Ainv = np.linlag.pinv(A)
-
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      +

      It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse. +It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices. +

      -

      Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD.

      +

      Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \))

      +$$ +\boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T, +$$ + +

      where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{\Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.

      @@ -337,32 +324,26 @@ The simple answer is to use the linear algebra function for pseudoinvers, that i 
      import numpy as np
 # SVD inversion
 def SVDinv(A):
-    ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
-    SVD is numerically more stable than the inversion algorithms provided by
-    numpy and scipy.linalg at the cost of being slower.
-    '''
     U, s, VT = np.linalg.svd(A)
-    print('test U')
-    print( (np.transpose(U) @ U - U @np.transpose(U)))
-    print('test VT')
-    print( (np.transpose(VT) @ VT - VT @np.transpose(VT)))
+    # reciprocals of singular values of s
+    d = 1.0 / s
+    # create m x n D matrix
+    D = np.zeros(A.shape)
+    # populate D with n x n diagonal matrix
+    D[:A.shape[1], :A.shape[1]] = np.diag(d)
+    UT = np.transpose(U)
+    V = np.transpose(VT)
+    return np.matmul(V,np.matmul(D.T,UT))
 
 
-    D = np.zeros((len(U),len(VT)))
-    D = np.diag(s)
-    UT = np.transpose(U); V = np.transpose(VT); invD = np.linalg.inv(D)
-    return np.matmul(V,np.matmul(invD,UT))
-
-
-#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])
-# Non-singular square matrix
-X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])
-print(X)
-A = np.transpose(X) @ X
-# Brute force inversion
-B = np.linalg.inv(A)  # here we could use np.linalg.pinv(A)
+A = np.array([ [0.3, 0.4], [0.5, 0.6], [0.7, 0.8],[0.9, 1.0]])
+print(A)
+# Brute force inversion of super-collinear matrix
+B = np.linalg.pinv(A)
+print(B)
+# Compare our own algorithm with pinv
 C = SVDinv(A)
-print(np.abs(B-C))
+print(np.abs(C-B))
      @@ -378,6 +359,7 @@ C = SVDinv(A) +

      As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by Numpy.

      @@ -401,7 +383,7 @@ C = SVDinv(A)

    757. 16
    758. 17
    759. ...
      760. -
    760. 60
      761. +
    761. 59
    762. »
      763. diff --git a/doc/pub/week36/html/._week36-bs008.html b/doc/pub/week36/html/._week36-bs008.html index a986fe0c1..ecd6ed89f 100644 --- a/doc/pub/week36/html/._week36-bs008.html +++ b/doc/pub/week36/html/._week36-bs008.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    763. Plans for week 36
      764. -
    764. Thursday September 8
      765. -
    765. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      766. -
    766. Linear Regression and the SVD
      767. -
    767. What does it mean?
      768. -
    768. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      769. -
    769. Code for SVD and Inversion of Matrices
      770. -
    770. Inverse of Rectangular Matrix
      771. -
    771. Ridge and LASSO Regression
      772. -
    772. From OLS to Ridge and Lasso
      773. -
    773. Deriving the Ridge Regression Equations
      774. -
    774. Note on Scikit-Learn
      775. -
    775. Comparison with OLS
      776. -
    776. SVD analysis
      777. -
    777. Interpreting the Ridge results
      778. -
    778. More interpretations
      779. -
    779. Deriving the Lasso Regression Equations
      780. -
    780. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      781. -
    781. Ridge Regression
      782. -
    782. Lasso Regression
      783. -
    783. Yet another Example
      784. -
    784. The OLS case
      785. -
    785. The Ridge case
      786. -
    786. Writing the Cost Function
      787. -
    787. Lasso case
      788. -
    788. The first Case
      789. -
    789. Simple code for solving the above problem
      790. -
    790. With Lasso Regression
      791. -
    791. Another Example, now with a polynomial fit
      792. -
    792. To think about, first part
      793. -
    793. More thinking
      794. -
    794. Still thinking
      795. -
    795. What does centering (subtracting the mean values) mean mathematically?
      796. -
    796. Further Manipulations
      797. -
    797. Wrapping it up
      798. -
    798. Linear Regression code, Intercept handling first
      799. -
    799. Code Examples
      800. -
    800. Taking out the mean
      801. -
    801. Friday September 9
      802. -
    802. Linking the regression analysis with a statistical interpretation
      803. -
    803. Assumptions made
      804. -
    804. Expectation value and variance
      805. -
    805. Expectation value and variance for \( \boldsymbol{\beta} \)
      806. -
    806. Deriving OLS from a probability distribution
      807. -
    807. Independent and Identically Distrubuted (iid)
      808. -
    808. Maximum Likelihood Estimation (MLE)
      809. -
    809. A new Cost Function
      810. -
    810. More basic Statistics and Bayes' theorem
      811. -
    811. Marginal Probability
      812. -
    812. Conditional Probability
      813. -
    813. Bayes' Theorem
      814. -
    814. Interpretations of Bayes' Theorem
      815. -
    815. Example of Usage of Bayes' theorem
      816. -
    816. Doing it correctly
      817. -
    817. Bayes' Theorem and Ridge and Lasso Regression
      818. -
    818. Test Function for what happens with OLS, Ridge and Lasso
      819. -
    819. Invoking Bayes' theorem
      820. -
    820. Ridge and Bayes
      821. -
    821. Lasso and Bayes
      822. -
    822. Exercise 1: mean values and variances in linear regression
      823. -
    823. Exercise 2: Adding Ridge and Lasso Regression
      824. +
    824. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      825. +
    825. Linear Regression and the SVD
      826. +
    826. What does it mean?
      827. +
    827. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      828. +
    828. Code for SVD and Inversion of Matrices
      829. +
    829. Inverse of Rectangular Matrix
      830. +
    830. Ridge and LASSO Regression
      831. +
    831. From OLS to Ridge and Lasso
      832. +
    832. Deriving the Ridge Regression Equations
      833. +
    833. Note on Scikit-Learn
      834. +
    834. Comparison with OLS
      835. +
    835. SVD analysis
      836. +
    836. Interpreting the Ridge results
      837. +
    837. More interpretations
      838. +
    838. Deriving the Lasso Regression Equations
      839. +
    839. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      840. +
    840. Ridge Regression
      841. +
    841. Lasso Regression
      842. +
    842. Yet another Example
      843. +
    843. The OLS case
      844. +
    844. The Ridge case
      845. +
    845. Writing the Cost Function
      846. +
    846. Lasso case
      847. +
    847. The first Case
      848. +
    848. Simple code for solving the above problem
      849. +
    849. With Lasso Regression
      850. +
    850. Another Example, now with a polynomial fit
      851. +
    851. To think about, first part
      852. +
    852. More thinking
      853. +
    853. Still thinking
      854. +
    854. What does centering (subtracting the mean values) mean mathematically?
      855. +
    855. Further Manipulations
      856. +
    856. Wrapping it up
      857. +
    857. Linear Regression code, Intercept handling first
      858. +
    858. Code Examples
      859. +
    859. Taking out the mean
      860. +
    860. Friday September 9
      861. +
    861. Linking the regression analysis with a statistical interpretation
      862. +
    862. Assumptions made
      863. +
    863. Expectation value and variance
      864. +
    864. Expectation value and variance for \( \boldsymbol{\beta} \)
      865. +
    865. Deriving OLS from a probability distribution
      866. +
    866. Independent and Identically Distrubuted (iid)
      867. +
    867. Maximum Likelihood Estimation (MLE)
      868. +
    868. A new Cost Function
      869. +
    869. More basic Statistics and Bayes' theorem
      870. +
    870. Marginal Probability
      871. +
    871. Conditional Probability
      872. +
    872. Bayes' Theorem
      873. +
    873. Interpretations of Bayes' Theorem
      874. +
    874. Example of Usage of Bayes' theorem
      875. +
    875. Doing it correctly
      876. +
    876. Bayes' Theorem and Ridge and Lasso Regression
      877. +
    877. Test Function for what happens with OLS, Ridge and Lasso
      878. +
    878. Invoking Bayes' theorem
      879. +
    879. Ridge and Bayes
      880. +
    880. Lasso and Bayes
      881. +
    881. Exercise 1: mean values and variances in linear regression
      882. +
    882. Exercise 2: Adding Ridge and Lasso Regression
      883. @@ -297,71 +295,26 @@ MathJax.Hub.Config({

       

       

       

      -

      Inverse of Rectangular Matrix

      +

      Ridge and LASSO Regression

      -

      Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular.

      - -

      The pseudoinverse is the generalization of the matrix inverse for square matrices to -rectangular matrices where the number of rows and columns are not equal. +

      Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is

      - -

      It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse. -It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices. -

      - -

      Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \))

      $$ -\boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T, +{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -

      where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{\Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.

      +

      or we can state it as

      +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +$$ +

      where we have used the definition of a norm-2 vector, that is

      +$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ - -
      -
      -
      -
      -
      - 
      import numpy as np
-# SVD inversion
-def SVDinv(A):
-    U, s, VT = np.linalg.svd(A)
-    # reciprocals of singular values of s
-    d = 1.0 / s
-    # create m x n D matrix
-    D = np.zeros(A.shape)
-    # populate D with n x n diagonal matrix
-    D[:A.shape[1], :A.shape[1]] = np.diag(d)
-    UT = np.transpose(U)
-    V = np.transpose(VT)
-    return np.matmul(V,np.matmul(D.T,UT))
-
-
-A = np.array([ [0.3, 0.4], [0.5, 0.6], [0.7, 0.8],[0.9, 1.0]])
-print(A)
-# Brute force inversion of super-collinear matrix
-B = np.linalg.pinv(A)
-print(B)
-# Compare our own algorithm with pinv
-C = SVDinv(A)
-print(np.abs(C-B))
-
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      - -

      As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by Numpy.

      @@ -386,7 +339,7 @@ C = SVDinv(A)

    883. 17
    884. 18
    885. ...
      886. -
    886. 60
      887. +
    887. 59
    888. »
      889. diff --git a/doc/pub/week36/html/._week36-bs009.html b/doc/pub/week36/html/._week36-bs009.html index 5ba117a00..561a90d58 100644 --- a/doc/pub/week36/html/._week36-bs009.html +++ b/doc/pub/week36/html/._week36-bs009.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    889. Plans for week 36
      890. -
    890. Thursday September 8
      891. -
    891. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      892. -
    892. Linear Regression and the SVD
      893. -
    893. What does it mean?
      894. -
    894. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      895. -
    895. Code for SVD and Inversion of Matrices
      896. -
    896. Inverse of Rectangular Matrix
      897. -
    897. Ridge and LASSO Regression
      898. -
    898. From OLS to Ridge and Lasso
      899. -
    899. Deriving the Ridge Regression Equations
      900. -
    900. Note on Scikit-Learn
      901. -
    901. Comparison with OLS
      902. -
    902. SVD analysis
      903. -
    903. Interpreting the Ridge results
      904. -
    904. More interpretations
      905. -
    905. Deriving the Lasso Regression Equations
      906. -
    906. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      907. -
    907. Ridge Regression
      908. -
    908. Lasso Regression
      909. -
    909. Yet another Example
      910. -
    910. The OLS case
      911. -
    911. The Ridge case
      912. -
    912. Writing the Cost Function
      913. -
    913. Lasso case
      914. -
    914. The first Case
      915. -
    915. Simple code for solving the above problem
      916. -
    916. With Lasso Regression
      917. -
    917. Another Example, now with a polynomial fit
      918. -
    918. To think about, first part
      919. -
    919. More thinking
      920. -
    920. Still thinking
      921. -
    921. What does centering (subtracting the mean values) mean mathematically?
      922. -
    922. Further Manipulations
      923. -
    923. Wrapping it up
      924. -
    924. Linear Regression code, Intercept handling first
      925. -
    925. Code Examples
      926. -
    926. Taking out the mean
      927. -
    927. Friday September 9
      928. -
    928. Linking the regression analysis with a statistical interpretation
      929. -
    929. Assumptions made
      930. -
    930. Expectation value and variance
      931. -
    931. Expectation value and variance for \( \boldsymbol{\beta} \)
      932. -
    932. Deriving OLS from a probability distribution
      933. -
    933. Independent and Identically Distrubuted (iid)
      934. -
    934. Maximum Likelihood Estimation (MLE)
      935. -
    935. A new Cost Function
      936. -
    936. More basic Statistics and Bayes' theorem
      937. -
    937. Marginal Probability
      938. -
    938. Conditional Probability
      939. -
    939. Bayes' Theorem
      940. -
    940. Interpretations of Bayes' Theorem
      941. -
    941. Example of Usage of Bayes' theorem
      942. -
    942. Doing it correctly
      943. -
    943. Bayes' Theorem and Ridge and Lasso Regression
      944. -
    944. Test Function for what happens with OLS, Ridge and Lasso
      945. -
    945. Invoking Bayes' theorem
      946. -
    946. Ridge and Bayes
      947. -
    947. Lasso and Bayes
      948. -
    948. Exercise 1: mean values and variances in linear regression
      949. -
    949. Exercise 2: Adding Ridge and Lasso Regression
      950. +
    950. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      951. +
    951. Linear Regression and the SVD
      952. +
    952. What does it mean?
      953. +
    953. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      954. +
    954. Code for SVD and Inversion of Matrices
      955. +
    955. Inverse of Rectangular Matrix
      956. +
    956. Ridge and LASSO Regression
      957. +
    957. From OLS to Ridge and Lasso
      958. +
    958. Deriving the Ridge Regression Equations
      959. +
    959. Note on Scikit-Learn
      960. +
    960. Comparison with OLS
      961. +
    961. SVD analysis
      962. +
    962. Interpreting the Ridge results
      963. +
    963. More interpretations
      964. +
    964. Deriving the Lasso Regression Equations
      965. +
    965. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      966. +
    966. Ridge Regression
      967. +
    967. Lasso Regression
      968. +
    968. Yet another Example
      969. +
    969. The OLS case
      970. +
    970. The Ridge case
      971. +
    971. Writing the Cost Function
      972. +
    972. Lasso case
      973. +
    973. The first Case
      974. +
    974. Simple code for solving the above problem
      975. +
    975. With Lasso Regression
      976. +
    976. Another Example, now with a polynomial fit
      977. +
    977. To think about, first part
      978. +
    978. More thinking
      979. +
    979. Still thinking
      980. +
    980. What does centering (subtracting the mean values) mean mathematically?
      981. +
    981. Further Manipulations
      982. +
    982. Wrapping it up
      983. +
    983. Linear Regression code, Intercept handling first
      984. +
    984. Code Examples
      985. +
    985. Taking out the mean
      986. +
    986. Friday September 9
      987. +
    987. Linking the regression analysis with a statistical interpretation
      988. +
    988. Assumptions made
      989. +
    989. Expectation value and variance
      990. +
    990. Expectation value and variance for \( \boldsymbol{\beta} \)
      991. +
    991. Deriving OLS from a probability distribution
      992. +
    992. Independent and Identically Distrubuted (iid)
      993. +
    993. Maximum Likelihood Estimation (MLE)
      994. +
    994. A new Cost Function
      995. +
    995. More basic Statistics and Bayes' theorem
      996. +
    996. Marginal Probability
      997. +
    997. Conditional Probability
      998. +
    998. Bayes' Theorem
      999. +
    999. Interpretations of Bayes' Theorem
      1000. +
    1000. Example of Usage of Bayes' theorem
      1001. +
    1001. Doing it correctly
      1002. +
    1002. Bayes' Theorem and Ridge and Lasso Regression
      1003. +
    1003. Test Function for what happens with OLS, Ridge and Lasso
      1004. +
    1004. Invoking Bayes' theorem
      1005. +
    1005. Ridge and Bayes
      1006. +
    1006. Lasso and Bayes
      1007. +
    1007. Exercise 1: mean values and variances in linear regression
      1008. +
    1008. Exercise 2: Adding Ridge and Lasso Regression
      1009. @@ -297,24 +295,39 @@ MathJax.Hub.Config({

       

       

       

      -

      Ridge and LASSO Regression

      +

      From OLS to Ridge and Lasso

      -

      Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is -our optimization problem is +

      By minimizing the above equation with respect to the parameters +\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is

      -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. -$$ -

      or we can state it as

      $$ {\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 $$ -

      where we have used the definition of a norm-2 vector, that is

      +

      which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining +

      + $$ -\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +$$ + +

      we have a new optimization equation

      +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +$$ + +

      which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

      + +

      Here we have defined the norm-1 as

      +$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. $$ @@ -342,7 +355,7 @@ $$
    1009. 18
    1010. 19
    1011. ...
      1012. -
    1012. 60
      1013. +
    1013. 59
    1014. »
      1015. diff --git a/doc/pub/week36/html/._week36-bs010.html b/doc/pub/week36/html/._week36-bs010.html index 07b8419a6..7e6571fdf 100644 --- a/doc/pub/week36/html/._week36-bs010.html +++ b/doc/pub/week36/html/._week36-bs010.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    1015. Plans for week 36
      1016. -
    1016. Thursday September 8
      1017. -
    1017. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1018. -
    1018. Linear Regression and the SVD
      1019. -
    1019. What does it mean?
      1020. -
    1020. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1021. -
    1021. Code for SVD and Inversion of Matrices
      1022. -
    1022. Inverse of Rectangular Matrix
      1023. -
    1023. Ridge and LASSO Regression
      1024. -
    1024. From OLS to Ridge and Lasso
      1025. -
    1025. Deriving the Ridge Regression Equations
      1026. -
    1026. Note on Scikit-Learn
      1027. -
    1027. Comparison with OLS
      1028. -
    1028. SVD analysis
      1029. -
    1029. Interpreting the Ridge results
      1030. -
    1030. More interpretations
      1031. -
    1031. Deriving the Lasso Regression Equations
      1032. -
    1032. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1033. -
    1033. Ridge Regression
      1034. -
    1034. Lasso Regression
      1035. -
    1035. Yet another Example
      1036. -
    1036. The OLS case
      1037. -
    1037. The Ridge case
      1038. -
    1038. Writing the Cost Function
      1039. -
    1039. Lasso case
      1040. -
    1040. The first Case
      1041. -
    1041. Simple code for solving the above problem
      1042. -
    1042. With Lasso Regression
      1043. -
    1043. Another Example, now with a polynomial fit
      1044. -
    1044. To think about, first part
      1045. -
    1045. More thinking
      1046. -
    1046. Still thinking
      1047. -
    1047. What does centering (subtracting the mean values) mean mathematically?
      1048. -
    1048. Further Manipulations
      1049. -
    1049. Wrapping it up
      1050. -
    1050. Linear Regression code, Intercept handling first
      1051. -
    1051. Code Examples
      1052. -
    1052. Taking out the mean
      1053. -
    1053. Friday September 9
      1054. -
    1054. Linking the regression analysis with a statistical interpretation
      1055. -
    1055. Assumptions made
      1056. -
    1056. Expectation value and variance
      1057. -
    1057. Expectation value and variance for \( \boldsymbol{\beta} \)
      1058. -
    1058. Deriving OLS from a probability distribution
      1059. -
    1059. Independent and Identically Distrubuted (iid)
      1060. -
    1060. Maximum Likelihood Estimation (MLE)
      1061. -
    1061. A new Cost Function
      1062. -
    1062. More basic Statistics and Bayes' theorem
      1063. -
    1063. Marginal Probability
      1064. -
    1064. Conditional Probability
      1065. -
    1065. Bayes' Theorem
      1066. -
    1066. Interpretations of Bayes' Theorem
      1067. -
    1067. Example of Usage of Bayes' theorem
      1068. -
    1068. Doing it correctly
      1069. -
    1069. Bayes' Theorem and Ridge and Lasso Regression
      1070. -
    1070. Test Function for what happens with OLS, Ridge and Lasso
      1071. -
    1071. Invoking Bayes' theorem
      1072. -
    1072. Ridge and Bayes
      1073. -
    1073. Lasso and Bayes
      1074. -
    1074. Exercise 1: mean values and variances in linear regression
      1075. -
    1075. Exercise 2: Adding Ridge and Lasso Regression
      1076. +
    1076. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1077. +
    1077. Linear Regression and the SVD
      1078. +
    1078. What does it mean?
      1079. +
    1079. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1080. +
    1080. Code for SVD and Inversion of Matrices
      1081. +
    1081. Inverse of Rectangular Matrix
      1082. +
    1082. Ridge and LASSO Regression
      1083. +
    1083. From OLS to Ridge and Lasso
      1084. +
    1084. Deriving the Ridge Regression Equations
      1085. +
    1085. Note on Scikit-Learn
      1086. +
    1086. Comparison with OLS
      1087. +
    1087. SVD analysis
      1088. +
    1088. Interpreting the Ridge results
      1089. +
    1089. More interpretations
      1090. +
    1090. Deriving the Lasso Regression Equations
      1091. +
    1091. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1092. +
    1092. Ridge Regression
      1093. +
    1093. Lasso Regression
      1094. +
    1094. Yet another Example
      1095. +
    1095. The OLS case
      1096. +
    1096. The Ridge case
      1097. +
    1097. Writing the Cost Function
      1098. +
    1098. Lasso case
      1099. +
    1099. The first Case
      1100. +
    1100. Simple code for solving the above problem
      1101. +
    1101. With Lasso Regression
      1102. +
    1102. Another Example, now with a polynomial fit
      1103. +
    1103. To think about, first part
      1104. +
    1104. More thinking
      1105. +
    1105. Still thinking
      1106. +
    1106. What does centering (subtracting the mean values) mean mathematically?
      1107. +
    1107. Further Manipulations
      1108. +
    1108. Wrapping it up
      1109. +
    1109. Linear Regression code, Intercept handling first
      1110. +
    1110. Code Examples
      1111. +
    1111. Taking out the mean
      1112. +
    1112. Friday September 9
      1113. +
    1113. Linking the regression analysis with a statistical interpretation
      1114. +
    1114. Assumptions made
      1115. +
    1115. Expectation value and variance
      1116. +
    1116. Expectation value and variance for \( \boldsymbol{\beta} \)
      1117. +
    1117. Deriving OLS from a probability distribution
      1118. +
    1118. Independent and Identically Distrubuted (iid)
      1119. +
    1119. Maximum Likelihood Estimation (MLE)
      1120. +
    1120. A new Cost Function
      1121. +
    1121. More basic Statistics and Bayes' theorem
      1122. +
    1122. Marginal Probability
      1123. +
    1123. Conditional Probability
      1124. +
    1124. Bayes' Theorem
      1125. +
    1125. Interpretations of Bayes' Theorem
      1126. +
    1126. Example of Usage of Bayes' theorem
      1127. +
    1127. Doing it correctly
      1128. +
    1128. Bayes' Theorem and Ridge and Lasso Regression
      1129. +
    1129. Test Function for what happens with OLS, Ridge and Lasso
      1130. +
    1130. Invoking Bayes' theorem
      1131. +
    1131. Ridge and Bayes
      1132. +
    1132. Lasso and Bayes
      1133. +
    1133. Exercise 1: mean values and variances in linear regression
      1134. +
    1134. Exercise 2: Adding Ridge and Lasso Regression
      1135. @@ -297,41 +295,31 @@ MathJax.Hub.Config({

       

       

       

      -

      From OLS to Ridge and Lasso

      +

      Deriving the Ridge Regression Equations

      -

      By minimizing the above equation with respect to the parameters -\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the -parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by -defining a new cost function to be optimized, that is +

      Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have

      + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +$$ + +

      and +taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain +the optimal parameters

      - $$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -

      which leads to the Ridge regression minimization problem where we -require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is -a finite number larger than zero. By defining -

      +

      with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that

      $$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, -$$ - -

      we have a new optimization equation

      -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 -$$ - -

      which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

      - -

      Here we have defined the norm-1 as

      -$$ -\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +\sum_{i=0}^{p-1} \beta_i^2 \leq t, $$ +

      with \( t \) a finite positive number.

      @@ -358,7 +346,7 @@ $$

    1135. 19
    1136. 20
    1137. ...
      1138. -
    1138. 60
      1139. +
    1139. 59
    1140. »
      1141. diff --git a/doc/pub/week36/html/._week36-bs011.html b/doc/pub/week36/html/._week36-bs011.html index 7e97353b7..899897538 100644 --- a/doc/pub/week36/html/._week36-bs011.html +++ b/doc/pub/week36/html/._week36-bs011.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    1141. Plans for week 36
      1142. -
    1142. Thursday September 8
      1143. -
    1143. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1144. -
    1144. Linear Regression and the SVD
      1145. -
    1145. What does it mean?
      1146. -
    1146. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1147. -
    1147. Code for SVD and Inversion of Matrices
      1148. -
    1148. Inverse of Rectangular Matrix
      1149. -
    1149. Ridge and LASSO Regression
      1150. -
    1150. From OLS to Ridge and Lasso
      1151. -
    1151. Deriving the Ridge Regression Equations
      1152. -
    1152. Note on Scikit-Learn
      1153. -
    1153. Comparison with OLS
      1154. -
    1154. SVD analysis
      1155. -
    1155. Interpreting the Ridge results
      1156. -
    1156. More interpretations
      1157. -
    1157. Deriving the Lasso Regression Equations
      1158. -
    1158. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1159. -
    1159. Ridge Regression
      1160. -
    1160. Lasso Regression
      1161. -
    1161. Yet another Example
      1162. -
    1162. The OLS case
      1163. -
    1163. The Ridge case
      1164. -
    1164. Writing the Cost Function
      1165. -
    1165. Lasso case
      1166. -
    1166. The first Case
      1167. -
    1167. Simple code for solving the above problem
      1168. -
    1168. With Lasso Regression
      1169. -
    1169. Another Example, now with a polynomial fit
      1170. -
    1170. To think about, first part
      1171. -
    1171. More thinking
      1172. -
    1172. Still thinking
      1173. -
    1173. What does centering (subtracting the mean values) mean mathematically?
      1174. -
    1174. Further Manipulations
      1175. -
    1175. Wrapping it up
      1176. -
    1176. Linear Regression code, Intercept handling first
      1177. -
    1177. Code Examples
      1178. -
    1178. Taking out the mean
      1179. -
    1179. Friday September 9
      1180. -
    1180. Linking the regression analysis with a statistical interpretation
      1181. -
    1181. Assumptions made
      1182. -
    1182. Expectation value and variance
      1183. -
    1183. Expectation value and variance for \( \boldsymbol{\beta} \)
      1184. -
    1184. Deriving OLS from a probability distribution
      1185. -
    1185. Independent and Identically Distrubuted (iid)
      1186. -
    1186. Maximum Likelihood Estimation (MLE)
      1187. -
    1187. A new Cost Function
      1188. -
    1188. More basic Statistics and Bayes' theorem
      1189. -
    1189. Marginal Probability
      1190. -
    1190. Conditional Probability
      1191. -
    1191. Bayes' Theorem
      1192. -
    1192. Interpretations of Bayes' Theorem
      1193. -
    1193. Example of Usage of Bayes' theorem
      1194. -
    1194. Doing it correctly
      1195. -
    1195. Bayes' Theorem and Ridge and Lasso Regression
      1196. -
    1196. Test Function for what happens with OLS, Ridge and Lasso
      1197. -
    1197. Invoking Bayes' theorem
      1198. -
    1198. Ridge and Bayes
      1199. -
    1199. Lasso and Bayes
      1200. -
    1200. Exercise 1: mean values and variances in linear regression
      1201. -
    1201. Exercise 2: Adding Ridge and Lasso Regression
      1202. +
    1202. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1203. +
    1203. Linear Regression and the SVD
      1204. +
    1204. What does it mean?
      1205. +
    1205. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1206. +
    1206. Code for SVD and Inversion of Matrices
      1207. +
    1207. Inverse of Rectangular Matrix
      1208. +
    1208. Ridge and LASSO Regression
      1209. +
    1209. From OLS to Ridge and Lasso
      1210. +
    1210. Deriving the Ridge Regression Equations
      1211. +
    1211. Note on Scikit-Learn
      1212. +
    1212. Comparison with OLS
      1213. +
    1213. SVD analysis
      1214. +
    1214. Interpreting the Ridge results
      1215. +
    1215. More interpretations
      1216. +
    1216. Deriving the Lasso Regression Equations
      1217. +
    1217. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1218. +
    1218. Ridge Regression
      1219. +
    1219. Lasso Regression
      1220. +
    1220. Yet another Example
      1221. +
    1221. The OLS case
      1222. +
    1222. The Ridge case
      1223. +
    1223. Writing the Cost Function
      1224. +
    1224. Lasso case
      1225. +
    1225. The first Case
      1226. +
    1226. Simple code for solving the above problem
      1227. +
    1227. With Lasso Regression
      1228. +
    1228. Another Example, now with a polynomial fit
      1229. +
    1229. To think about, first part
      1230. +
    1230. More thinking
      1231. +
    1231. Still thinking
      1232. +
    1232. What does centering (subtracting the mean values) mean mathematically?
      1233. +
    1233. Further Manipulations
      1234. +
    1234. Wrapping it up
      1235. +
    1235. Linear Regression code, Intercept handling first
      1236. +
    1236. Code Examples
      1237. +
    1237. Taking out the mean
      1238. +
    1238. Friday September 9
      1239. +
    1239. Linking the regression analysis with a statistical interpretation
      1240. +
    1240. Assumptions made
      1241. +
    1241. Expectation value and variance
      1242. +
    1242. Expectation value and variance for \( \boldsymbol{\beta} \)
      1243. +
    1243. Deriving OLS from a probability distribution
      1244. +
    1244. Independent and Identically Distrubuted (iid)
      1245. +
    1245. Maximum Likelihood Estimation (MLE)
      1246. +
    1246. A new Cost Function
      1247. +
    1247. More basic Statistics and Bayes' theorem
      1248. +
    1248. Marginal Probability
      1249. +
    1249. Conditional Probability
      1250. +
    1250. Bayes' Theorem
      1251. +
    1251. Interpretations of Bayes' Theorem
      1252. +
    1252. Example of Usage of Bayes' theorem
      1253. +
    1253. Doing it correctly
      1254. +
    1254. Bayes' Theorem and Ridge and Lasso Regression
      1255. +
    1255. Test Function for what happens with OLS, Ridge and Lasso
      1256. +
    1256. Invoking Bayes' theorem
      1257. +
    1257. Ridge and Bayes
      1258. +
    1258. Lasso and Bayes
      1259. +
    1259. Exercise 1: mean values and variances in linear regression
      1260. +
    1260. Exercise 2: Adding Ridge and Lasso Regression
      1261. @@ -297,31 +295,15 @@ MathJax.Hub.Config({

       

       

       

      -

      Deriving the Ridge Regression Equations

      +

      Note on Scikit-Learn

      -

      Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have

      +

      Note well that a library like Scikit-Learn does not include the \( 1/n \) factor in the expression for the mean-squared error. If you include it, the optimal parameter \( \beta \) becomes

      $$ -C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

      and -taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then -a slightly modified matrix inversion problem which for finite values -of \( \lambda \) does not suffer from singularity problems. We obtain -the optimal parameters -

      -$$ -\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ - -

      with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that

      - -$$ -\sum_{i=0}^{p-1} \beta_i^2 \leq t, -$$ - -

      with \( t \) a finite positive number.

      +

      In our codes where we compare our own codes with Scikit-Learn, we do thus not include the \( 1/n \) factor in the cost function.

      @@ -348,7 +330,7 @@ $$

    1261. 20
    1262. 21
    1263. ...
      1264. -
    1264. 60
      1265. +
    1265. 59
    1266. »
      1267. diff --git a/doc/pub/week36/html/._week36-bs012.html b/doc/pub/week36/html/._week36-bs012.html index 3698108e4..2b98d7995 100644 --- a/doc/pub/week36/html/._week36-bs012.html +++ b/doc/pub/week36/html/._week36-bs012.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    1267. Plans for week 36
      1268. -
    1268. Thursday September 8
      1269. -
    1269. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1270. -
    1270. Linear Regression and the SVD
      1271. -
    1271. What does it mean?
      1272. -
    1272. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1273. -
    1273. Code for SVD and Inversion of Matrices
      1274. -
    1274. Inverse of Rectangular Matrix
      1275. -
    1275. Ridge and LASSO Regression
      1276. -
    1276. From OLS to Ridge and Lasso
      1277. -
    1277. Deriving the Ridge Regression Equations
      1278. -
    1278. Note on Scikit-Learn
      1279. -
    1279. Comparison with OLS
      1280. -
    1280. SVD analysis
      1281. -
    1281. Interpreting the Ridge results
      1282. -
    1282. More interpretations
      1283. -
    1283. Deriving the Lasso Regression Equations
      1284. -
    1284. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1285. -
    1285. Ridge Regression
      1286. -
    1286. Lasso Regression
      1287. -
    1287. Yet another Example
      1288. -
    1288. The OLS case
      1289. -
    1289. The Ridge case
      1290. -
    1290. Writing the Cost Function
      1291. -
    1291. Lasso case
      1292. -
    1292. The first Case
      1293. -
    1293. Simple code for solving the above problem
      1294. -
    1294. With Lasso Regression
      1295. -
    1295. Another Example, now with a polynomial fit
      1296. -
    1296. To think about, first part
      1297. -
    1297. More thinking
      1298. -
    1298. Still thinking
      1299. -
    1299. What does centering (subtracting the mean values) mean mathematically?
      1300. -
    1300. Further Manipulations
      1301. -
    1301. Wrapping it up
      1302. -
    1302. Linear Regression code, Intercept handling first
      1303. -
    1303. Code Examples
      1304. -
    1304. Taking out the mean
      1305. -
    1305. Friday September 9
      1306. -
    1306. Linking the regression analysis with a statistical interpretation
      1307. -
    1307. Assumptions made
      1308. -
    1308. Expectation value and variance
      1309. -
    1309. Expectation value and variance for \( \boldsymbol{\beta} \)
      1310. -
    1310. Deriving OLS from a probability distribution
      1311. -
    1311. Independent and Identically Distrubuted (iid)
      1312. -
    1312. Maximum Likelihood Estimation (MLE)
      1313. -
    1313. A new Cost Function
      1314. -
    1314. More basic Statistics and Bayes' theorem
      1315. -
    1315. Marginal Probability
      1316. -
    1316. Conditional Probability
      1317. -
    1317. Bayes' Theorem
      1318. -
    1318. Interpretations of Bayes' Theorem
      1319. -
    1319. Example of Usage of Bayes' theorem
      1320. -
    1320. Doing it correctly
      1321. -
    1321. Bayes' Theorem and Ridge and Lasso Regression
      1322. -
    1322. Test Function for what happens with OLS, Ridge and Lasso
      1323. -
    1323. Invoking Bayes' theorem
      1324. -
    1324. Ridge and Bayes
      1325. -
    1325. Lasso and Bayes
      1326. -
    1326. Exercise 1: mean values and variances in linear regression
      1327. -
    1327. Exercise 2: Adding Ridge and Lasso Regression
      1328. +
    1328. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1329. +
    1329. Linear Regression and the SVD
      1330. +
    1330. What does it mean?
      1331. +
    1331. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1332. +
    1332. Code for SVD and Inversion of Matrices
      1333. +
    1333. Inverse of Rectangular Matrix
      1334. +
    1334. Ridge and LASSO Regression
      1335. +
    1335. From OLS to Ridge and Lasso
      1336. +
    1336. Deriving the Ridge Regression Equations
      1337. +
    1337. Note on Scikit-Learn
      1338. +
    1338. Comparison with OLS
      1339. +
    1339. SVD analysis
      1340. +
    1340. Interpreting the Ridge results
      1341. +
    1341. More interpretations
      1342. +
    1342. Deriving the Lasso Regression Equations
      1343. +
    1343. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1344. +
    1344. Ridge Regression
      1345. +
    1345. Lasso Regression
      1346. +
    1346. Yet another Example
      1347. +
    1347. The OLS case
      1348. +
    1348. The Ridge case
      1349. +
    1349. Writing the Cost Function
      1350. +
    1350. Lasso case
      1351. +
    1351. The first Case
      1352. +
    1352. Simple code for solving the above problem
      1353. +
    1353. With Lasso Regression
      1354. +
    1354. Another Example, now with a polynomial fit
      1355. +
    1355. To think about, first part
      1356. +
    1356. More thinking
      1357. +
    1357. Still thinking
      1358. +
    1358. What does centering (subtracting the mean values) mean mathematically?
      1359. +
    1359. Further Manipulations
      1360. +
    1360. Wrapping it up
      1361. +
    1361. Linear Regression code, Intercept handling first
      1362. +
    1362. Code Examples
      1363. +
    1363. Taking out the mean
      1364. +
    1364. Friday September 9
      1365. +
    1365. Linking the regression analysis with a statistical interpretation
      1366. +
    1366. Assumptions made
      1367. +
    1367. Expectation value and variance
      1368. +
    1368. Expectation value and variance for \( \boldsymbol{\beta} \)
      1369. +
    1369. Deriving OLS from a probability distribution
      1370. +
    1370. Independent and Identically Distrubuted (iid)
      1371. +
    1371. Maximum Likelihood Estimation (MLE)
      1372. +
    1372. A new Cost Function
      1373. +
    1373. More basic Statistics and Bayes' theorem
      1374. +
    1374. Marginal Probability
      1375. +
    1375. Conditional Probability
      1376. +
    1376. Bayes' Theorem
      1377. +
    1377. Interpretations of Bayes' Theorem
      1378. +
    1378. Example of Usage of Bayes' theorem
      1379. +
    1379. Doing it correctly
      1380. +
    1380. Bayes' Theorem and Ridge and Lasso Regression
      1381. +
    1381. Test Function for what happens with OLS, Ridge and Lasso
      1382. +
    1382. Invoking Bayes' theorem
      1383. +
    1383. Ridge and Bayes
      1384. +
    1384. Lasso and Bayes
      1385. +
    1385. Exercise 1: mean values and variances in linear regression
      1386. +
    1386. Exercise 2: Adding Ridge and Lasso Regression
      1387. @@ -297,15 +295,20 @@ MathJax.Hub.Config({

       

       

       

      -

      Note on Scikit-Learn

      - -

      Note well that a library like Scikit-Learn does not include the \( 1/n \) factor in the expression for the mean-squared error. If you include it, the optimal parameter \( \beta \) becomes

      - +

      Comparison with OLS

      +

      When we compare this with the ordinary least squares result we have

      $$ -\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -

      In our codes where we compare our own codes with Scikit-Learn, we do thus not include the \( 1/n \) factor in the cost function.

      +

      which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).

      + +

      We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \( \lambda \), we may +even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

      @@ -332,7 +335,7 @@ $$

    1387. 21
    1388. 22
    1389. ...
      1390. -
    1390. 60
      1391. +
    1391. 59
    1392. »
      1393. diff --git a/doc/pub/week36/html/._week36-bs013.html b/doc/pub/week36/html/._week36-bs013.html index e97a461b5..66d60dfed 100644 --- a/doc/pub/week36/html/._week36-bs013.html +++ b/doc/pub/week36/html/._week36-bs013.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    1393. Plans for week 36
      1394. -
    1394. Thursday September 8
      1395. -
    1395. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1396. -
    1396. Linear Regression and the SVD
      1397. -
    1397. What does it mean?
      1398. -
    1398. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1399. -
    1399. Code for SVD and Inversion of Matrices
      1400. -
    1400. Inverse of Rectangular Matrix
      1401. -
    1401. Ridge and LASSO Regression
      1402. -
    1402. From OLS to Ridge and Lasso
      1403. -
    1403. Deriving the Ridge Regression Equations
      1404. -
    1404. Note on Scikit-Learn
      1405. -
    1405. Comparison with OLS
      1406. -
    1406. SVD analysis
      1407. -
    1407. Interpreting the Ridge results
      1408. -
    1408. More interpretations
      1409. -
    1409. Deriving the Lasso Regression Equations
      1410. -
    1410. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1411. -
    1411. Ridge Regression
      1412. -
    1412. Lasso Regression
      1413. -
    1413. Yet another Example
      1414. -
    1414. The OLS case
      1415. -
    1415. The Ridge case
      1416. -
    1416. Writing the Cost Function
      1417. -
    1417. Lasso case
      1418. -
    1418. The first Case
      1419. -
    1419. Simple code for solving the above problem
      1420. -
    1420. With Lasso Regression
      1421. -
    1421. Another Example, now with a polynomial fit
      1422. -
    1422. To think about, first part
      1423. -
    1423. More thinking
      1424. -
    1424. Still thinking
      1425. -
    1425. What does centering (subtracting the mean values) mean mathematically?
      1426. -
    1426. Further Manipulations
      1427. -
    1427. Wrapping it up
      1428. -
    1428. Linear Regression code, Intercept handling first
      1429. -
    1429. Code Examples
      1430. -
    1430. Taking out the mean
      1431. -
    1431. Friday September 9
      1432. -
    1432. Linking the regression analysis with a statistical interpretation
      1433. -
    1433. Assumptions made
      1434. -
    1434. Expectation value and variance
      1435. -
    1435. Expectation value and variance for \( \boldsymbol{\beta} \)
      1436. -
    1436. Deriving OLS from a probability distribution
      1437. -
    1437. Independent and Identically Distrubuted (iid)
      1438. -
    1438. Maximum Likelihood Estimation (MLE)
      1439. -
    1439. A new Cost Function
      1440. -
    1440. More basic Statistics and Bayes' theorem
      1441. -
    1441. Marginal Probability
      1442. -
    1442. Conditional Probability
      1443. -
    1443. Bayes' Theorem
      1444. -
    1444. Interpretations of Bayes' Theorem
      1445. -
    1445. Example of Usage of Bayes' theorem
      1446. -
    1446. Doing it correctly
      1447. -
    1447. Bayes' Theorem and Ridge and Lasso Regression
      1448. -
    1448. Test Function for what happens with OLS, Ridge and Lasso
      1449. -
    1449. Invoking Bayes' theorem
      1450. -
    1450. Ridge and Bayes
      1451. -
    1451. Lasso and Bayes
      1452. -
    1452. Exercise 1: mean values and variances in linear regression
      1453. -
    1453. Exercise 2: Adding Ridge and Lasso Regression
      1454. +
    1454. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1455. +
    1455. Linear Regression and the SVD
      1456. +
    1456. What does it mean?
      1457. +
    1457. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1458. +
    1458. Code for SVD and Inversion of Matrices
      1459. +
    1459. Inverse of Rectangular Matrix
      1460. +
    1460. Ridge and LASSO Regression
      1461. +
    1461. From OLS to Ridge and Lasso
      1462. +
    1462. Deriving the Ridge Regression Equations
      1463. +
    1463. Note on Scikit-Learn
      1464. +
    1464. Comparison with OLS
      1465. +
    1465. SVD analysis
      1466. +
    1466. Interpreting the Ridge results
      1467. +
    1467. More interpretations
      1468. +
    1468. Deriving the Lasso Regression Equations
      1469. +
    1469. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1470. +
    1470. Ridge Regression
      1471. +
    1471. Lasso Regression
      1472. +
    1472. Yet another Example
      1473. +
    1473. The OLS case
      1474. +
    1474. The Ridge case
      1475. +
    1475. Writing the Cost Function
      1476. +
    1476. Lasso case
      1477. +
    1477. The first Case
      1478. +
    1478. Simple code for solving the above problem
      1479. +
    1479. With Lasso Regression
      1480. +
    1480. Another Example, now with a polynomial fit
      1481. +
    1481. To think about, first part
      1482. +
    1482. More thinking
      1483. +
    1483. Still thinking
      1484. +
    1484. What does centering (subtracting the mean values) mean mathematically?
      1485. +
    1485. Further Manipulations
      1486. +
    1486. Wrapping it up
      1487. +
    1487. Linear Regression code, Intercept handling first
      1488. +
    1488. Code Examples
      1489. +
    1489. Taking out the mean
      1490. +
    1490. Friday September 9
      1491. +
    1491. Linking the regression analysis with a statistical interpretation
      1492. +
    1492. Assumptions made
      1493. +
    1493. Expectation value and variance
      1494. +
    1494. Expectation value and variance for \( \boldsymbol{\beta} \)
      1495. +
    1495. Deriving OLS from a probability distribution
      1496. +
    1496. Independent and Identically Distrubuted (iid)
      1497. +
    1497. Maximum Likelihood Estimation (MLE)
      1498. +
    1498. A new Cost Function
      1499. +
    1499. More basic Statistics and Bayes' theorem
      1500. +
    1500. Marginal Probability
      1501. +
    1501. Conditional Probability
      1502. +
    1502. Bayes' Theorem
      1503. +
    1503. Interpretations of Bayes' Theorem
      1504. +
    1504. Example of Usage of Bayes' theorem
      1505. +
    1505. Doing it correctly
      1506. +
    1506. Bayes' Theorem and Ridge and Lasso Regression
      1507. +
    1507. Test Function for what happens with OLS, Ridge and Lasso
      1508. +
    1508. Invoking Bayes' theorem
      1509. +
    1509. Ridge and Bayes
      1510. +
    1510. Lasso and Bayes
      1511. +
    1511. Exercise 1: mean values and variances in linear regression
      1512. +
    1512. Exercise 2: Adding Ridge and Lasso Regression
      1513. @@ -297,20 +295,22 @@ MathJax.Hub.Config({

       

       

       

      -

      Comparison with OLS

      -

      When we compare this with the ordinary least squares result we have

      -$$ -\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ +

      SVD analysis

      -

      which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).

      - -

      We see that Ridge regression is nothing but the standard OLS with a -modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in -particular for our discussion of the bias-variance tradeoff are rather -interesting. We will see that for specific values of \( \lambda \), we may -even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

      Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as

      +$$ +\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. +$$ + +

      For Ridge regression this becomes

      + +$$ +\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +$$ + +

      with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).

      @@ -337,7 +337,7 @@ even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). The

    1513. 22
    1514. 23
    1515. ...
      1516. -
    1516. 60
      1517. +
    1517. 59
    1518. »
      1519. diff --git a/doc/pub/week36/html/._week36-bs014.html b/doc/pub/week36/html/._week36-bs014.html index 093030d9d..5b829507f 100644 --- a/doc/pub/week36/html/._week36-bs014.html +++ b/doc/pub/week36/html/._week36-bs014.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    1519. Plans for week 36
      1520. -
    1520. Thursday September 8
      1521. -
    1521. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1522. -
    1522. Linear Regression and the SVD
      1523. -
    1523. What does it mean?
      1524. -
    1524. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1525. -
    1525. Code for SVD and Inversion of Matrices
      1526. -
    1526. Inverse of Rectangular Matrix
      1527. -
    1527. Ridge and LASSO Regression
      1528. -
    1528. From OLS to Ridge and Lasso
      1529. -
    1529. Deriving the Ridge Regression Equations
      1530. -
    1530. Note on Scikit-Learn
      1531. -
    1531. Comparison with OLS
      1532. -
    1532. SVD analysis
      1533. -
    1533. Interpreting the Ridge results
      1534. -
    1534. More interpretations
      1535. -
    1535. Deriving the Lasso Regression Equations
      1536. -
    1536. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1537. -
    1537. Ridge Regression
      1538. -
    1538. Lasso Regression
      1539. -
    1539. Yet another Example
      1540. -
    1540. The OLS case
      1541. -
    1541. The Ridge case
      1542. -
    1542. Writing the Cost Function
      1543. -
    1543. Lasso case
      1544. -
    1544. The first Case
      1545. -
    1545. Simple code for solving the above problem
      1546. -
    1546. With Lasso Regression
      1547. -
    1547. Another Example, now with a polynomial fit
      1548. -
    1548. To think about, first part
      1549. -
    1549. More thinking
      1550. -
    1550. Still thinking
      1551. -
    1551. What does centering (subtracting the mean values) mean mathematically?
      1552. -
    1552. Further Manipulations
      1553. -
    1553. Wrapping it up
      1554. -
    1554. Linear Regression code, Intercept handling first
      1555. -
    1555. Code Examples
      1556. -
    1556. Taking out the mean
      1557. -
    1557. Friday September 9
      1558. -
    1558. Linking the regression analysis with a statistical interpretation
      1559. -
    1559. Assumptions made
      1560. -
    1560. Expectation value and variance
      1561. -
    1561. Expectation value and variance for \( \boldsymbol{\beta} \)
      1562. -
    1562. Deriving OLS from a probability distribution
      1563. -
    1563. Independent and Identically Distrubuted (iid)
      1564. -
    1564. Maximum Likelihood Estimation (MLE)
      1565. -
    1565. A new Cost Function
      1566. -
    1566. More basic Statistics and Bayes' theorem
      1567. -
    1567. Marginal Probability
      1568. -
    1568. Conditional Probability
      1569. -
    1569. Bayes' Theorem
      1570. -
    1570. Interpretations of Bayes' Theorem
      1571. -
    1571. Example of Usage of Bayes' theorem
      1572. -
    1572. Doing it correctly
      1573. -
    1573. Bayes' Theorem and Ridge and Lasso Regression
      1574. -
    1574. Test Function for what happens with OLS, Ridge and Lasso
      1575. -
    1575. Invoking Bayes' theorem
      1576. -
    1576. Ridge and Bayes
      1577. -
    1577. Lasso and Bayes
      1578. -
    1578. Exercise 1: mean values and variances in linear regression
      1579. -
    1579. Exercise 2: Adding Ridge and Lasso Regression
      1580. +
    1580. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1581. +
    1581. Linear Regression and the SVD
      1582. +
    1582. What does it mean?
      1583. +
    1583. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1584. +
    1584. Code for SVD and Inversion of Matrices
      1585. +
    1585. Inverse of Rectangular Matrix
      1586. +
    1586. Ridge and LASSO Regression
      1587. +
    1587. From OLS to Ridge and Lasso
      1588. +
    1588. Deriving the Ridge Regression Equations
      1589. +
    1589. Note on Scikit-Learn
      1590. +
    1590. Comparison with OLS
      1591. +
    1591. SVD analysis
      1592. +
    1592. Interpreting the Ridge results
      1593. +
    1593. More interpretations
      1594. +
    1594. Deriving the Lasso Regression Equations
      1595. +
    1595. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1596. +
    1596. Ridge Regression
      1597. +
    1597. Lasso Regression
      1598. +
    1598. Yet another Example
      1599. +
    1599. The OLS case
      1600. +
    1600. The Ridge case
      1601. +
    1601. Writing the Cost Function
      1602. +
    1602. Lasso case
      1603. +
    1603. The first Case
      1604. +
    1604. Simple code for solving the above problem
      1605. +
    1605. With Lasso Regression
      1606. +
    1606. Another Example, now with a polynomial fit
      1607. +
    1607. To think about, first part
      1608. +
    1608. More thinking
      1609. +
    1609. Still thinking
      1610. +
    1610. What does centering (subtracting the mean values) mean mathematically?
      1611. +
    1611. Further Manipulations
      1612. +
    1612. Wrapping it up
      1613. +
    1613. Linear Regression code, Intercept handling first
      1614. +
    1614. Code Examples
      1615. +
    1615. Taking out the mean
      1616. +
    1616. Friday September 9
      1617. +
    1617. Linking the regression analysis with a statistical interpretation
      1618. +
    1618. Assumptions made
      1619. +
    1619. Expectation value and variance
      1620. +
    1620. Expectation value and variance for \( \boldsymbol{\beta} \)
      1621. +
    1621. Deriving OLS from a probability distribution
      1622. +
    1622. Independent and Identically Distrubuted (iid)
      1623. +
    1623. Maximum Likelihood Estimation (MLE)
      1624. +
    1624. A new Cost Function
      1625. +
    1625. More basic Statistics and Bayes' theorem
      1626. +
    1626. Marginal Probability
      1627. +
    1627. Conditional Probability
      1628. +
    1628. Bayes' Theorem
      1629. +
    1629. Interpretations of Bayes' Theorem
      1630. +
    1630. Example of Usage of Bayes' theorem
      1631. +
    1631. Doing it correctly
      1632. +
    1632. Bayes' Theorem and Ridge and Lasso Regression
      1633. +
    1633. Test Function for what happens with OLS, Ridge and Lasso
      1634. +
    1634. Invoking Bayes' theorem
      1635. +
    1635. Ridge and Bayes
      1636. +
    1636. Lasso and Bayes
      1637. +
    1637. Exercise 1: mean values and variances in linear regression
      1638. +
    1638. Exercise 2: Adding Ridge and Lasso Regression
      1639. @@ -297,22 +295,22 @@ MathJax.Hub.Config({

       

       

       

      -

      SVD analysis

      +

      Interpreting the Ridge results

      -

      Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) -We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +

      Since \( \lambda \geq 0 \), it means that compared to OLS, we have

      + +$$ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +$$ + +

      Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \).

      -$$ -\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. -$$ -

      For Ridge regression this becomes

      - -$$ -\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, -$$ - -

      with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).

      +

      For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.

      @@ -339,7 +337,7 @@ $$

    1639. 23
    1640. 24
    1641. ...
      1642. -
    1642. 60
      1643. +
    1643. 59
    1644. »
      1645. diff --git a/doc/pub/week36/html/._week36-bs015.html b/doc/pub/week36/html/._week36-bs015.html index f7cd403cf..e4c997e23 100644 --- a/doc/pub/week36/html/._week36-bs015.html +++ b/doc/pub/week36/html/._week36-bs015.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    1645. Plans for week 36
      1646. -
    1646. Thursday September 8
      1647. -
    1647. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1648. -
    1648. Linear Regression and the SVD
      1649. -
    1649. What does it mean?
      1650. -
    1650. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1651. -
    1651. Code for SVD and Inversion of Matrices
      1652. -
    1652. Inverse of Rectangular Matrix
      1653. -
    1653. Ridge and LASSO Regression
      1654. -
    1654. From OLS to Ridge and Lasso
      1655. -
    1655. Deriving the Ridge Regression Equations
      1656. -
    1656. Note on Scikit-Learn
      1657. -
    1657. Comparison with OLS
      1658. -
    1658. SVD analysis
      1659. -
    1659. Interpreting the Ridge results
      1660. -
    1660. More interpretations
      1661. -
    1661. Deriving the Lasso Regression Equations
      1662. -
    1662. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1663. -
    1663. Ridge Regression
      1664. -
    1664. Lasso Regression
      1665. -
    1665. Yet another Example
      1666. -
    1666. The OLS case
      1667. -
    1667. The Ridge case
      1668. -
    1668. Writing the Cost Function
      1669. -
    1669. Lasso case
      1670. -
    1670. The first Case
      1671. -
    1671. Simple code for solving the above problem
      1672. -
    1672. With Lasso Regression
      1673. -
    1673. Another Example, now with a polynomial fit
      1674. -
    1674. To think about, first part
      1675. -
    1675. More thinking
      1676. -
    1676. Still thinking
      1677. -
    1677. What does centering (subtracting the mean values) mean mathematically?
      1678. -
    1678. Further Manipulations
      1679. -
    1679. Wrapping it up
      1680. -
    1680. Linear Regression code, Intercept handling first
      1681. -
    1681. Code Examples
      1682. -
    1682. Taking out the mean
      1683. -
    1683. Friday September 9
      1684. -
    1684. Linking the regression analysis with a statistical interpretation
      1685. -
    1685. Assumptions made
      1686. -
    1686. Expectation value and variance
      1687. -
    1687. Expectation value and variance for \( \boldsymbol{\beta} \)
      1688. -
    1688. Deriving OLS from a probability distribution
      1689. -
    1689. Independent and Identically Distrubuted (iid)
      1690. -
    1690. Maximum Likelihood Estimation (MLE)
      1691. -
    1691. A new Cost Function
      1692. -
    1692. More basic Statistics and Bayes' theorem
      1693. -
    1693. Marginal Probability
      1694. -
    1694. Conditional Probability
      1695. -
    1695. Bayes' Theorem
      1696. -
    1696. Interpretations of Bayes' Theorem
      1697. -
    1697. Example of Usage of Bayes' theorem
      1698. -
    1698. Doing it correctly
      1699. -
    1699. Bayes' Theorem and Ridge and Lasso Regression
      1700. -
    1700. Test Function for what happens with OLS, Ridge and Lasso
      1701. -
    1701. Invoking Bayes' theorem
      1702. -
    1702. Ridge and Bayes
      1703. -
    1703. Lasso and Bayes
      1704. -
    1704. Exercise 1: mean values and variances in linear regression
      1705. -
    1705. Exercise 2: Adding Ridge and Lasso Regression
      1706. +
    1706. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1707. +
    1707. Linear Regression and the SVD
      1708. +
    1708. What does it mean?
      1709. +
    1709. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1710. +
    1710. Code for SVD and Inversion of Matrices
      1711. +
    1711. Inverse of Rectangular Matrix
      1712. +
    1712. Ridge and LASSO Regression
      1713. +
    1713. From OLS to Ridge and Lasso
      1714. +
    1714. Deriving the Ridge Regression Equations
      1715. +
    1715. Note on Scikit-Learn
      1716. +
    1716. Comparison with OLS
      1717. +
    1717. SVD analysis
      1718. +
    1718. Interpreting the Ridge results
      1719. +
    1719. More interpretations
      1720. +
    1720. Deriving the Lasso Regression Equations
      1721. +
    1721. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1722. +
    1722. Ridge Regression
      1723. +
    1723. Lasso Regression
      1724. +
    1724. Yet another Example
      1725. +
    1725. The OLS case
      1726. +
    1726. The Ridge case
      1727. +
    1727. Writing the Cost Function
      1728. +
    1728. Lasso case
      1729. +
    1729. The first Case
      1730. +
    1730. Simple code for solving the above problem
      1731. +
    1731. With Lasso Regression
      1732. +
    1732. Another Example, now with a polynomial fit
      1733. +
    1733. To think about, first part
      1734. +
    1734. More thinking
      1735. +
    1735. Still thinking
      1736. +
    1736. What does centering (subtracting the mean values) mean mathematically?
      1737. +
    1737. Further Manipulations
      1738. +
    1738. Wrapping it up
      1739. +
    1739. Linear Regression code, Intercept handling first
      1740. +
    1740. Code Examples
      1741. +
    1741. Taking out the mean
      1742. +
    1742. Friday September 9
      1743. +
    1743. Linking the regression analysis with a statistical interpretation
      1744. +
    1744. Assumptions made
      1745. +
    1745. Expectation value and variance
      1746. +
    1746. Expectation value and variance for \( \boldsymbol{\beta} \)
      1747. +
    1747. Deriving OLS from a probability distribution
      1748. +
    1748. Independent and Identically Distrubuted (iid)
      1749. +
    1749. Maximum Likelihood Estimation (MLE)
      1750. +
    1750. A new Cost Function
      1751. +
    1751. More basic Statistics and Bayes' theorem
      1752. +
    1752. Marginal Probability
      1753. +
    1753. Conditional Probability
      1754. +
    1754. Bayes' Theorem
      1755. +
    1755. Interpretations of Bayes' Theorem
      1756. +
    1756. Example of Usage of Bayes' theorem
      1757. +
    1757. Doing it correctly
      1758. +
    1758. Bayes' Theorem and Ridge and Lasso Regression
      1759. +
    1759. Test Function for what happens with OLS, Ridge and Lasso
      1760. +
    1760. Invoking Bayes' theorem
      1761. +
    1761. Ridge and Bayes
      1762. +
    1762. Lasso and Bayes
      1763. +
    1763. Exercise 1: mean values and variances in linear regression
      1764. +
    1764. Exercise 2: Adding Ridge and Lasso Regression
      1765. @@ -297,22 +295,35 @@ MathJax.Hub.Config({

       

       

       

      -

      Interpreting the Ridge results

      +

      More interpretations

      -

      Since \( \lambda \geq 0 \), it means that compared to OLS, we have

      +

      For the sake of simplicity, let us assume that the design matrix is orthonormal, that is

      $$ -\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. $$ -

      Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the -orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by -\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has -eigenvalues ordered in a descending way, that is \( \sigma_i \geq -\sigma_{i+1} \). +

      In this case the standard OLS results in

      +$$ +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y}, +$$ + +

      and

      + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +$$ + +

      that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity.

      -

      For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.

      +

      We will come back to more interpreations after we have gone through some of the statistical analysis part.

      + +

      For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. +

      @@ -339,7 +350,7 @@ eigenvalues ordered in a descending way, that is \( \sigma_i \geq

    1765. 24
    1766. 25
    1767. ...
      1768. -
    1768. 60
      1769. +
    1769. 59
    1770. »
      1771. diff --git a/doc/pub/week36/html/._week36-bs016.html b/doc/pub/week36/html/._week36-bs016.html index ceee78588..e914af0f0 100644 --- a/doc/pub/week36/html/._week36-bs016.html +++ b/doc/pub/week36/html/._week36-bs016.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    1771. Plans for week 36
      1772. -
    1772. Thursday September 8
      1773. -
    1773. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1774. -
    1774. Linear Regression and the SVD
      1775. -
    1775. What does it mean?
      1776. -
    1776. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1777. -
    1777. Code for SVD and Inversion of Matrices
      1778. -
    1778. Inverse of Rectangular Matrix
      1779. -
    1779. Ridge and LASSO Regression
      1780. -
    1780. From OLS to Ridge and Lasso
      1781. -
    1781. Deriving the Ridge Regression Equations
      1782. -
    1782. Note on Scikit-Learn
      1783. -
    1783. Comparison with OLS
      1784. -
    1784. SVD analysis
      1785. -
    1785. Interpreting the Ridge results
      1786. -
    1786. More interpretations
      1787. -
    1787. Deriving the Lasso Regression Equations
      1788. -
    1788. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1789. -
    1789. Ridge Regression
      1790. -
    1790. Lasso Regression
      1791. -
    1791. Yet another Example
      1792. -
    1792. The OLS case
      1793. -
    1793. The Ridge case
      1794. -
    1794. Writing the Cost Function
      1795. -
    1795. Lasso case
      1796. -
    1796. The first Case
      1797. -
    1797. Simple code for solving the above problem
      1798. -
    1798. With Lasso Regression
      1799. -
    1799. Another Example, now with a polynomial fit
      1800. -
    1800. To think about, first part
      1801. -
    1801. More thinking
      1802. -
    1802. Still thinking
      1803. -
    1803. What does centering (subtracting the mean values) mean mathematically?
      1804. -
    1804. Further Manipulations
      1805. -
    1805. Wrapping it up
      1806. -
    1806. Linear Regression code, Intercept handling first
      1807. -
    1807. Code Examples
      1808. -
    1808. Taking out the mean
      1809. -
    1809. Friday September 9
      1810. -
    1810. Linking the regression analysis with a statistical interpretation
      1811. -
    1811. Assumptions made
      1812. -
    1812. Expectation value and variance
      1813. -
    1813. Expectation value and variance for \( \boldsymbol{\beta} \)
      1814. -
    1814. Deriving OLS from a probability distribution
      1815. -
    1815. Independent and Identically Distrubuted (iid)
      1816. -
    1816. Maximum Likelihood Estimation (MLE)
      1817. -
    1817. A new Cost Function
      1818. -
    1818. More basic Statistics and Bayes' theorem
      1819. -
    1819. Marginal Probability
      1820. -
    1820. Conditional Probability
      1821. -
    1821. Bayes' Theorem
      1822. -
    1822. Interpretations of Bayes' Theorem
      1823. -
    1823. Example of Usage of Bayes' theorem
      1824. -
    1824. Doing it correctly
      1825. -
    1825. Bayes' Theorem and Ridge and Lasso Regression
      1826. -
    1826. Test Function for what happens with OLS, Ridge and Lasso
      1827. -
    1827. Invoking Bayes' theorem
      1828. -
    1828. Ridge and Bayes
      1829. -
    1829. Lasso and Bayes
      1830. -
    1830. Exercise 1: mean values and variances in linear regression
      1831. -
    1831. Exercise 2: Adding Ridge and Lasso Regression
      1832. +
    1832. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1833. +
    1833. Linear Regression and the SVD
      1834. +
    1834. What does it mean?
      1835. +
    1835. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1836. +
    1836. Code for SVD and Inversion of Matrices
      1837. +
    1837. Inverse of Rectangular Matrix
      1838. +
    1838. Ridge and LASSO Regression
      1839. +
    1839. From OLS to Ridge and Lasso
      1840. +
    1840. Deriving the Ridge Regression Equations
      1841. +
    1841. Note on Scikit-Learn
      1842. +
    1842. Comparison with OLS
      1843. +
    1843. SVD analysis
      1844. +
    1844. Interpreting the Ridge results
      1845. +
    1845. More interpretations
      1846. +
    1846. Deriving the Lasso Regression Equations
      1847. +
    1847. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1848. +
    1848. Ridge Regression
      1849. +
    1849. Lasso Regression
      1850. +
    1850. Yet another Example
      1851. +
    1851. The OLS case
      1852. +
    1852. The Ridge case
      1853. +
    1853. Writing the Cost Function
      1854. +
    1854. Lasso case
      1855. +
    1855. The first Case
      1856. +
    1856. Simple code for solving the above problem
      1857. +
    1857. With Lasso Regression
      1858. +
    1858. Another Example, now with a polynomial fit
      1859. +
    1859. To think about, first part
      1860. +
    1860. More thinking
      1861. +
    1861. Still thinking
      1862. +
    1862. What does centering (subtracting the mean values) mean mathematically?
      1863. +
    1863. Further Manipulations
      1864. +
    1864. Wrapping it up
      1865. +
    1865. Linear Regression code, Intercept handling first
      1866. +
    1866. Code Examples
      1867. +
    1867. Taking out the mean
      1868. +
    1868. Friday September 9
      1869. +
    1869. Linking the regression analysis with a statistical interpretation
      1870. +
    1870. Assumptions made
      1871. +
    1871. Expectation value and variance
      1872. +
    1872. Expectation value and variance for \( \boldsymbol{\beta} \)
      1873. +
    1873. Deriving OLS from a probability distribution
      1874. +
    1874. Independent and Identically Distrubuted (iid)
      1875. +
    1875. Maximum Likelihood Estimation (MLE)
      1876. +
    1876. A new Cost Function
      1877. +
    1877. More basic Statistics and Bayes' theorem
      1878. +
    1878. Marginal Probability
      1879. +
    1879. Conditional Probability
      1880. +
    1880. Bayes' Theorem
      1881. +
    1881. Interpretations of Bayes' Theorem
      1882. +
    1882. Example of Usage of Bayes' theorem
      1883. +
    1883. Doing it correctly
      1884. +
    1884. Bayes' Theorem and Ridge and Lasso Regression
      1885. +
    1885. Test Function for what happens with OLS, Ridge and Lasso
      1886. +
    1886. Invoking Bayes' theorem
      1887. +
    1887. Ridge and Bayes
      1888. +
    1888. Lasso and Bayes
      1889. +
    1889. Exercise 1: mean values and variances in linear regression
      1890. +
    1890. Exercise 2: Adding Ridge and Lasso Regression
      1891. @@ -297,35 +295,31 @@ MathJax.Hub.Config({

       

       

       

      -

      More interpretations

      +

      Deriving the Lasso Regression Equations

      -

      For the sake of simplicity, let us assume that the design matrix is orthonormal, that is

      +

      Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function

      $$ -\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. +C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -

      In this case the standard OLS results in

      +

      Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

      $$ -\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y}, +\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right. $$ -

      and

      +

      we have that the derivative of the cost function is

      $$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, $$ -

      that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and -the Ridge estimator converges to zero when the hyperparameter goes to -infinity. -

      +

      and reordering we have

      +$$ +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. +$$ -

      We will come back to more interpreations after we have gone through some of the statistical analysis part.

      - -

      For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. -Similarly, Mehta et al's article is also recommended. -

      +

      This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

      @@ -352,7 +346,7 @@ Similarly, Mehta et al

    1891. 25
    1892. 26
    1893. ...
      1894. -
    1894. 60
      1895. +
    1895. 59
    1896. »
      1897. diff --git a/doc/pub/week36/html/._week36-bs017.html b/doc/pub/week36/html/._week36-bs017.html index 0bfe5df46..43a007fee 100644 --- a/doc/pub/week36/html/._week36-bs017.html +++ b/doc/pub/week36/html/._week36-bs017.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    1897. Plans for week 36
      1898. -
    1898. Thursday September 8
      1899. -
    1899. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1900. -
    1900. Linear Regression and the SVD
      1901. -
    1901. What does it mean?
      1902. -
    1902. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1903. -
    1903. Code for SVD and Inversion of Matrices
      1904. -
    1904. Inverse of Rectangular Matrix
      1905. -
    1905. Ridge and LASSO Regression
      1906. -
    1906. From OLS to Ridge and Lasso
      1907. -
    1907. Deriving the Ridge Regression Equations
      1908. -
    1908. Note on Scikit-Learn
      1909. -
    1909. Comparison with OLS
      1910. -
    1910. SVD analysis
      1911. -
    1911. Interpreting the Ridge results
      1912. -
    1912. More interpretations
      1913. -
    1913. Deriving the Lasso Regression Equations
      1914. -
    1914. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1915. -
    1915. Ridge Regression
      1916. -
    1916. Lasso Regression
      1917. -
    1917. Yet another Example
      1918. -
    1918. The OLS case
      1919. -
    1919. The Ridge case
      1920. -
    1920. Writing the Cost Function
      1921. -
    1921. Lasso case
      1922. -
    1922. The first Case
      1923. -
    1923. Simple code for solving the above problem
      1924. -
    1924. With Lasso Regression
      1925. -
    1925. Another Example, now with a polynomial fit
      1926. -
    1926. To think about, first part
      1927. -
    1927. More thinking
      1928. -
    1928. Still thinking
      1929. -
    1929. What does centering (subtracting the mean values) mean mathematically?
      1930. -
    1930. Further Manipulations
      1931. -
    1931. Wrapping it up
      1932. -
    1932. Linear Regression code, Intercept handling first
      1933. -
    1933. Code Examples
      1934. -
    1934. Taking out the mean
      1935. -
    1935. Friday September 9
      1936. -
    1936. Linking the regression analysis with a statistical interpretation
      1937. -
    1937. Assumptions made
      1938. -
    1938. Expectation value and variance
      1939. -
    1939. Expectation value and variance for \( \boldsymbol{\beta} \)
      1940. -
    1940. Deriving OLS from a probability distribution
      1941. -
    1941. Independent and Identically Distrubuted (iid)
      1942. -
    1942. Maximum Likelihood Estimation (MLE)
      1943. -
    1943. A new Cost Function
      1944. -
    1944. More basic Statistics and Bayes' theorem
      1945. -
    1945. Marginal Probability
      1946. -
    1946. Conditional Probability
      1947. -
    1947. Bayes' Theorem
      1948. -
    1948. Interpretations of Bayes' Theorem
      1949. -
    1949. Example of Usage of Bayes' theorem
      1950. -
    1950. Doing it correctly
      1951. -
    1951. Bayes' Theorem and Ridge and Lasso Regression
      1952. -
    1952. Test Function for what happens with OLS, Ridge and Lasso
      1953. -
    1953. Invoking Bayes' theorem
      1954. -
    1954. Ridge and Bayes
      1955. -
    1955. Lasso and Bayes
      1956. -
    1956. Exercise 1: mean values and variances in linear regression
      1957. -
    1957. Exercise 2: Adding Ridge and Lasso Regression
      1958. +
    1958. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      1959. +
    1959. Linear Regression and the SVD
      1960. +
    1960. What does it mean?
      1961. +
    1961. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      1962. +
    1962. Code for SVD and Inversion of Matrices
      1963. +
    1963. Inverse of Rectangular Matrix
      1964. +
    1964. Ridge and LASSO Regression
      1965. +
    1965. From OLS to Ridge and Lasso
      1966. +
    1966. Deriving the Ridge Regression Equations
      1967. +
    1967. Note on Scikit-Learn
      1968. +
    1968. Comparison with OLS
      1969. +
    1969. SVD analysis
      1970. +
    1970. Interpreting the Ridge results
      1971. +
    1971. More interpretations
      1972. +
    1972. Deriving the Lasso Regression Equations
      1973. +
    1973. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      1974. +
    1974. Ridge Regression
      1975. +
    1975. Lasso Regression
      1976. +
    1976. Yet another Example
      1977. +
    1977. The OLS case
      1978. +
    1978. The Ridge case
      1979. +
    1979. Writing the Cost Function
      1980. +
    1980. Lasso case
      1981. +
    1981. The first Case
      1982. +
    1982. Simple code for solving the above problem
      1983. +
    1983. With Lasso Regression
      1984. +
    1984. Another Example, now with a polynomial fit
      1985. +
    1985. To think about, first part
      1986. +
    1986. More thinking
      1987. +
    1987. Still thinking
      1988. +
    1988. What does centering (subtracting the mean values) mean mathematically?
      1989. +
    1989. Further Manipulations
      1990. +
    1990. Wrapping it up
      1991. +
    1991. Linear Regression code, Intercept handling first
      1992. +
    1992. Code Examples
      1993. +
    1993. Taking out the mean
      1994. +
    1994. Friday September 9
      1995. +
    1995. Linking the regression analysis with a statistical interpretation
      1996. +
    1996. Assumptions made
      1997. +
    1997. Expectation value and variance
      1998. +
    1998. Expectation value and variance for \( \boldsymbol{\beta} \)
      1999. +
    1999. Deriving OLS from a probability distribution
      2000. +
    2000. Independent and Identically Distrubuted (iid)
      2001. +
    2001. Maximum Likelihood Estimation (MLE)
      2002. +
    2002. A new Cost Function
      2003. +
    2003. More basic Statistics and Bayes' theorem
      2004. +
    2004. Marginal Probability
      2005. +
    2005. Conditional Probability
      2006. +
    2006. Bayes' Theorem
      2007. +
    2007. Interpretations of Bayes' Theorem
      2008. +
    2008. Example of Usage of Bayes' theorem
      2009. +
    2009. Doing it correctly
      2010. +
    2010. Bayes' Theorem and Ridge and Lasso Regression
      2011. +
    2011. Test Function for what happens with OLS, Ridge and Lasso
      2012. +
    2012. Invoking Bayes' theorem
      2013. +
    2013. Ridge and Bayes
      2014. +
    2014. Lasso and Bayes
      2015. +
    2015. Exercise 1: mean values and variances in linear regression
      2016. +
    2016. Exercise 2: Adding Ridge and Lasso Regression
      2017. @@ -297,31 +295,22 @@ MathJax.Hub.Config({

       

       

       

      -

      Deriving the Lasso Regression Equations

      +

      Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression

      -

      Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function

      +

      Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the +diagonal. In this case we have an equal number of rows and columns \( n=p \). +

      +

      Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))

      $$ -C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, $$ -

      Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

      +

      and minimizing we have that

      $$ -\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right. +\hat{\beta}_i^{\mathrm{OLS}} = y_i. $$ -

      we have that the derivative of the cost function is

      - -$$ -\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, -$$ - -

      and reordering we have

      -$$ -\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. -$$ - -

      This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

      @@ -348,7 +337,7 @@ $$

    2017. 26
    2018. 27
    2019. ...
      2020. -
    2020. 60
      2021. +
    2021. 59
    2022. »
      2023. diff --git a/doc/pub/week36/html/._week36-bs018.html b/doc/pub/week36/html/._week36-bs018.html index 7da57dcd9..12a02f185 100644 --- a/doc/pub/week36/html/._week36-bs018.html +++ b/doc/pub/week36/html/._week36-bs018.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    2023. Plans for week 36
      2024. -
    2024. Thursday September 8
      2025. -
    2025. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2026. -
    2026. Linear Regression and the SVD
      2027. -
    2027. What does it mean?
      2028. -
    2028. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2029. -
    2029. Code for SVD and Inversion of Matrices
      2030. -
    2030. Inverse of Rectangular Matrix
      2031. -
    2031. Ridge and LASSO Regression
      2032. -
    2032. From OLS to Ridge and Lasso
      2033. -
    2033. Deriving the Ridge Regression Equations
      2034. -
    2034. Note on Scikit-Learn
      2035. -
    2035. Comparison with OLS
      2036. -
    2036. SVD analysis
      2037. -
    2037. Interpreting the Ridge results
      2038. -
    2038. More interpretations
      2039. -
    2039. Deriving the Lasso Regression Equations
      2040. -
    2040. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2041. -
    2041. Ridge Regression
      2042. -
    2042. Lasso Regression
      2043. -
    2043. Yet another Example
      2044. -
    2044. The OLS case
      2045. -
    2045. The Ridge case
      2046. -
    2046. Writing the Cost Function
      2047. -
    2047. Lasso case
      2048. -
    2048. The first Case
      2049. -
    2049. Simple code for solving the above problem
      2050. -
    2050. With Lasso Regression
      2051. -
    2051. Another Example, now with a polynomial fit
      2052. -
    2052. To think about, first part
      2053. -
    2053. More thinking
      2054. -
    2054. Still thinking
      2055. -
    2055. What does centering (subtracting the mean values) mean mathematically?
      2056. -
    2056. Further Manipulations
      2057. -
    2057. Wrapping it up
      2058. -
    2058. Linear Regression code, Intercept handling first
      2059. -
    2059. Code Examples
      2060. -
    2060. Taking out the mean
      2061. -
    2061. Friday September 9
      2062. -
    2062. Linking the regression analysis with a statistical interpretation
      2063. -
    2063. Assumptions made
      2064. -
    2064. Expectation value and variance
      2065. -
    2065. Expectation value and variance for \( \boldsymbol{\beta} \)
      2066. -
    2066. Deriving OLS from a probability distribution
      2067. -
    2067. Independent and Identically Distrubuted (iid)
      2068. -
    2068. Maximum Likelihood Estimation (MLE)
      2069. -
    2069. A new Cost Function
      2070. -
    2070. More basic Statistics and Bayes' theorem
      2071. -
    2071. Marginal Probability
      2072. -
    2072. Conditional Probability
      2073. -
    2073. Bayes' Theorem
      2074. -
    2074. Interpretations of Bayes' Theorem
      2075. -
    2075. Example of Usage of Bayes' theorem
      2076. -
    2076. Doing it correctly
      2077. -
    2077. Bayes' Theorem and Ridge and Lasso Regression
      2078. -
    2078. Test Function for what happens with OLS, Ridge and Lasso
      2079. -
    2079. Invoking Bayes' theorem
      2080. -
    2080. Ridge and Bayes
      2081. -
    2081. Lasso and Bayes
      2082. -
    2082. Exercise 1: mean values and variances in linear regression
      2083. -
    2083. Exercise 2: Adding Ridge and Lasso Regression
      2084. +
    2084. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2085. +
    2085. Linear Regression and the SVD
      2086. +
    2086. What does it mean?
      2087. +
    2087. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2088. +
    2088. Code for SVD and Inversion of Matrices
      2089. +
    2089. Inverse of Rectangular Matrix
      2090. +
    2090. Ridge and LASSO Regression
      2091. +
    2091. From OLS to Ridge and Lasso
      2092. +
    2092. Deriving the Ridge Regression Equations
      2093. +
    2093. Note on Scikit-Learn
      2094. +
    2094. Comparison with OLS
      2095. +
    2095. SVD analysis
      2096. +
    2096. Interpreting the Ridge results
      2097. +
    2097. More interpretations
      2098. +
    2098. Deriving the Lasso Regression Equations
      2099. +
    2099. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2100. +
    2100. Ridge Regression
      2101. +
    2101. Lasso Regression
      2102. +
    2102. Yet another Example
      2103. +
    2103. The OLS case
      2104. +
    2104. The Ridge case
      2105. +
    2105. Writing the Cost Function
      2106. +
    2106. Lasso case
      2107. +
    2107. The first Case
      2108. +
    2108. Simple code for solving the above problem
      2109. +
    2109. With Lasso Regression
      2110. +
    2110. Another Example, now with a polynomial fit
      2111. +
    2111. To think about, first part
      2112. +
    2112. More thinking
      2113. +
    2113. Still thinking
      2114. +
    2114. What does centering (subtracting the mean values) mean mathematically?
      2115. +
    2115. Further Manipulations
      2116. +
    2116. Wrapping it up
      2117. +
    2117. Linear Regression code, Intercept handling first
      2118. +
    2118. Code Examples
      2119. +
    2119. Taking out the mean
      2120. +
    2120. Friday September 9
      2121. +
    2121. Linking the regression analysis with a statistical interpretation
      2122. +
    2122. Assumptions made
      2123. +
    2123. Expectation value and variance
      2124. +
    2124. Expectation value and variance for \( \boldsymbol{\beta} \)
      2125. +
    2125. Deriving OLS from a probability distribution
      2126. +
    2126. Independent and Identically Distrubuted (iid)
      2127. +
    2127. Maximum Likelihood Estimation (MLE)
      2128. +
    2128. A new Cost Function
      2129. +
    2129. More basic Statistics and Bayes' theorem
      2130. +
    2130. Marginal Probability
      2131. +
    2131. Conditional Probability
      2132. +
    2132. Bayes' Theorem
      2133. +
    2133. Interpretations of Bayes' Theorem
      2134. +
    2134. Example of Usage of Bayes' theorem
      2135. +
    2135. Doing it correctly
      2136. +
    2136. Bayes' Theorem and Ridge and Lasso Regression
      2137. +
    2137. Test Function for what happens with OLS, Ridge and Lasso
      2138. +
    2138. Invoking Bayes' theorem
      2139. +
    2139. Ridge and Bayes
      2140. +
    2140. Lasso and Bayes
      2141. +
    2141. Exercise 1: mean values and variances in linear regression
      2142. +
    2142. Exercise 2: Adding Ridge and Lasso Regression
      2143. @@ -297,20 +295,16 @@ MathJax.Hub.Config({

       

       

       

      -

      Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression

      +

      Ridge Regression

      -

      Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the -diagonal. In this case we have an equal number of rows and columns \( n=p \). -

      - -

      Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))

      +

      For Ridge regression our cost function is

      $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$

      and minimizing we have that

      $$ -\hat{\beta}_i^{\mathrm{OLS}} = y_i. +\hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. $$ @@ -339,7 +333,7 @@ $$
    2143. 27
    2144. 28
    2145. ...
      2146. -
    2146. 60
      2147. +
    2147. 59
    2148. »
      2149. diff --git a/doc/pub/week36/html/._week36-bs019.html b/doc/pub/week36/html/._week36-bs019.html index e06dbdf14..3589e887f 100644 --- a/doc/pub/week36/html/._week36-bs019.html +++ b/doc/pub/week36/html/._week36-bs019.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    2149. Plans for week 36
      2150. -
    2150. Thursday September 8
      2151. -
    2151. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2152. -
    2152. Linear Regression and the SVD
      2153. -
    2153. What does it mean?
      2154. -
    2154. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2155. -
    2155. Code for SVD and Inversion of Matrices
      2156. -
    2156. Inverse of Rectangular Matrix
      2157. -
    2157. Ridge and LASSO Regression
      2158. -
    2158. From OLS to Ridge and Lasso
      2159. -
    2159. Deriving the Ridge Regression Equations
      2160. -
    2160. Note on Scikit-Learn
      2161. -
    2161. Comparison with OLS
      2162. -
    2162. SVD analysis
      2163. -
    2163. Interpreting the Ridge results
      2164. -
    2164. More interpretations
      2165. -
    2165. Deriving the Lasso Regression Equations
      2166. -
    2166. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2167. -
    2167. Ridge Regression
      2168. -
    2168. Lasso Regression
      2169. -
    2169. Yet another Example
      2170. -
    2170. The OLS case
      2171. -
    2171. The Ridge case
      2172. -
    2172. Writing the Cost Function
      2173. -
    2173. Lasso case
      2174. -
    2174. The first Case
      2175. -
    2175. Simple code for solving the above problem
      2176. -
    2176. With Lasso Regression
      2177. -
    2177. Another Example, now with a polynomial fit
      2178. -
    2178. To think about, first part
      2179. -
    2179. More thinking
      2180. -
    2180. Still thinking
      2181. -
    2181. What does centering (subtracting the mean values) mean mathematically?
      2182. -
    2182. Further Manipulations
      2183. -
    2183. Wrapping it up
      2184. -
    2184. Linear Regression code, Intercept handling first
      2185. -
    2185. Code Examples
      2186. -
    2186. Taking out the mean
      2187. -
    2187. Friday September 9
      2188. -
    2188. Linking the regression analysis with a statistical interpretation
      2189. -
    2189. Assumptions made
      2190. -
    2190. Expectation value and variance
      2191. -
    2191. Expectation value and variance for \( \boldsymbol{\beta} \)
      2192. -
    2192. Deriving OLS from a probability distribution
      2193. -
    2193. Independent and Identically Distrubuted (iid)
      2194. -
    2194. Maximum Likelihood Estimation (MLE)
      2195. -
    2195. A new Cost Function
      2196. -
    2196. More basic Statistics and Bayes' theorem
      2197. -
    2197. Marginal Probability
      2198. -
    2198. Conditional Probability
      2199. -
    2199. Bayes' Theorem
      2200. -
    2200. Interpretations of Bayes' Theorem
      2201. -
    2201. Example of Usage of Bayes' theorem
      2202. -
    2202. Doing it correctly
      2203. -
    2203. Bayes' Theorem and Ridge and Lasso Regression
      2204. -
    2204. Test Function for what happens with OLS, Ridge and Lasso
      2205. -
    2205. Invoking Bayes' theorem
      2206. -
    2206. Ridge and Bayes
      2207. -
    2207. Lasso and Bayes
      2208. -
    2208. Exercise 1: mean values and variances in linear regression
      2209. -
    2209. Exercise 2: Adding Ridge and Lasso Regression
      2210. +
    2210. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2211. +
    2211. Linear Regression and the SVD
      2212. +
    2212. What does it mean?
      2213. +
    2213. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2214. +
    2214. Code for SVD and Inversion of Matrices
      2215. +
    2215. Inverse of Rectangular Matrix
      2216. +
    2216. Ridge and LASSO Regression
      2217. +
    2217. From OLS to Ridge and Lasso
      2218. +
    2218. Deriving the Ridge Regression Equations
      2219. +
    2219. Note on Scikit-Learn
      2220. +
    2220. Comparison with OLS
      2221. +
    2221. SVD analysis
      2222. +
    2222. Interpreting the Ridge results
      2223. +
    2223. More interpretations
      2224. +
    2224. Deriving the Lasso Regression Equations
      2225. +
    2225. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2226. +
    2226. Ridge Regression
      2227. +
    2227. Lasso Regression
      2228. +
    2228. Yet another Example
      2229. +
    2229. The OLS case
      2230. +
    2230. The Ridge case
      2231. +
    2231. Writing the Cost Function
      2232. +
    2232. Lasso case
      2233. +
    2233. The first Case
      2234. +
    2234. Simple code for solving the above problem
      2235. +
    2235. With Lasso Regression
      2236. +
    2236. Another Example, now with a polynomial fit
      2237. +
    2237. To think about, first part
      2238. +
    2238. More thinking
      2239. +
    2239. Still thinking
      2240. +
    2240. What does centering (subtracting the mean values) mean mathematically?
      2241. +
    2241. Further Manipulations
      2242. +
    2242. Wrapping it up
      2243. +
    2243. Linear Regression code, Intercept handling first
      2244. +
    2244. Code Examples
      2245. +
    2245. Taking out the mean
      2246. +
    2246. Friday September 9
      2247. +
    2247. Linking the regression analysis with a statistical interpretation
      2248. +
    2248. Assumptions made
      2249. +
    2249. Expectation value and variance
      2250. +
    2250. Expectation value and variance for \( \boldsymbol{\beta} \)
      2251. +
    2251. Deriving OLS from a probability distribution
      2252. +
    2252. Independent and Identically Distrubuted (iid)
      2253. +
    2253. Maximum Likelihood Estimation (MLE)
      2254. +
    2254. A new Cost Function
      2255. +
    2255. More basic Statistics and Bayes' theorem
      2256. +
    2256. Marginal Probability
      2257. +
    2257. Conditional Probability
      2258. +
    2258. Bayes' Theorem
      2259. +
    2259. Interpretations of Bayes' Theorem
      2260. +
    2260. Example of Usage of Bayes' theorem
      2261. +
    2261. Doing it correctly
      2262. +
    2262. Bayes' Theorem and Ridge and Lasso Regression
      2263. +
    2263. Test Function for what happens with OLS, Ridge and Lasso
      2264. +
    2264. Invoking Bayes' theorem
      2265. +
    2265. Ridge and Bayes
      2266. +
    2266. Lasso and Bayes
      2267. +
    2267. Exercise 1: mean values and variances in linear regression
      2268. +
    2268. Exercise 2: Adding Ridge and Lasso Regression
      2269. @@ -297,18 +295,26 @@ MathJax.Hub.Config({

       

       

       

      -

      Ridge Regression

      +

      Lasso Regression

      -

      For Ridge regression our cost function is

      +

      For Lasso regression our cost function is

      $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, $$

      and minimizing we have that

      $$ -\hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. +-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, $$ +

      which leads to

      +$$ +\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ + y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ + 0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. +$$ + +

      Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).

      @@ -335,7 +341,7 @@ $$

    2269. 28
    2270. 29
    2271. ...
      2272. -
    2272. 60
      2273. +
    2273. 59
    2274. »
      2275. diff --git a/doc/pub/week36/html/._week36-bs020.html b/doc/pub/week36/html/._week36-bs020.html index 97357fbec..1158ac94e 100644 --- a/doc/pub/week36/html/._week36-bs020.html +++ b/doc/pub/week36/html/._week36-bs020.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    2275. Plans for week 36
      2276. -
    2276. Thursday September 8
      2277. -
    2277. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2278. -
    2278. Linear Regression and the SVD
      2279. -
    2279. What does it mean?
      2280. -
    2280. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2281. -
    2281. Code for SVD and Inversion of Matrices
      2282. -
    2282. Inverse of Rectangular Matrix
      2283. -
    2283. Ridge and LASSO Regression
      2284. -
    2284. From OLS to Ridge and Lasso
      2285. -
    2285. Deriving the Ridge Regression Equations
      2286. -
    2286. Note on Scikit-Learn
      2287. -
    2287. Comparison with OLS
      2288. -
    2288. SVD analysis
      2289. -
    2289. Interpreting the Ridge results
      2290. -
    2290. More interpretations
      2291. -
    2291. Deriving the Lasso Regression Equations
      2292. -
    2292. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2293. -
    2293. Ridge Regression
      2294. -
    2294. Lasso Regression
      2295. -
    2295. Yet another Example
      2296. -
    2296. The OLS case
      2297. -
    2297. The Ridge case
      2298. -
    2298. Writing the Cost Function
      2299. -
    2299. Lasso case
      2300. -
    2300. The first Case
      2301. -
    2301. Simple code for solving the above problem
      2302. -
    2302. With Lasso Regression
      2303. -
    2303. Another Example, now with a polynomial fit
      2304. -
    2304. To think about, first part
      2305. -
    2305. More thinking
      2306. -
    2306. Still thinking
      2307. -
    2307. What does centering (subtracting the mean values) mean mathematically?
      2308. -
    2308. Further Manipulations
      2309. -
    2309. Wrapping it up
      2310. -
    2310. Linear Regression code, Intercept handling first
      2311. -
    2311. Code Examples
      2312. -
    2312. Taking out the mean
      2313. -
    2313. Friday September 9
      2314. -
    2314. Linking the regression analysis with a statistical interpretation
      2315. -
    2315. Assumptions made
      2316. -
    2316. Expectation value and variance
      2317. -
    2317. Expectation value and variance for \( \boldsymbol{\beta} \)
      2318. -
    2318. Deriving OLS from a probability distribution
      2319. -
    2319. Independent and Identically Distrubuted (iid)
      2320. -
    2320. Maximum Likelihood Estimation (MLE)
      2321. -
    2321. A new Cost Function
      2322. -
    2322. More basic Statistics and Bayes' theorem
      2323. -
    2323. Marginal Probability
      2324. -
    2324. Conditional Probability
      2325. -
    2325. Bayes' Theorem
      2326. -
    2326. Interpretations of Bayes' Theorem
      2327. -
    2327. Example of Usage of Bayes' theorem
      2328. -
    2328. Doing it correctly
      2329. -
    2329. Bayes' Theorem and Ridge and Lasso Regression
      2330. -
    2330. Test Function for what happens with OLS, Ridge and Lasso
      2331. -
    2331. Invoking Bayes' theorem
      2332. -
    2332. Ridge and Bayes
      2333. -
    2333. Lasso and Bayes
      2334. -
    2334. Exercise 1: mean values and variances in linear regression
      2335. -
    2335. Exercise 2: Adding Ridge and Lasso Regression
      2336. +
    2336. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2337. +
    2337. Linear Regression and the SVD
      2338. +
    2338. What does it mean?
      2339. +
    2339. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2340. +
    2340. Code for SVD and Inversion of Matrices
      2341. +
    2341. Inverse of Rectangular Matrix
      2342. +
    2342. Ridge and LASSO Regression
      2343. +
    2343. From OLS to Ridge and Lasso
      2344. +
    2344. Deriving the Ridge Regression Equations
      2345. +
    2345. Note on Scikit-Learn
      2346. +
    2346. Comparison with OLS
      2347. +
    2347. SVD analysis
      2348. +
    2348. Interpreting the Ridge results
      2349. +
    2349. More interpretations
      2350. +
    2350. Deriving the Lasso Regression Equations
      2351. +
    2351. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2352. +
    2352. Ridge Regression
      2353. +
    2353. Lasso Regression
      2354. +
    2354. Yet another Example
      2355. +
    2355. The OLS case
      2356. +
    2356. The Ridge case
      2357. +
    2357. Writing the Cost Function
      2358. +
    2358. Lasso case
      2359. +
    2359. The first Case
      2360. +
    2360. Simple code for solving the above problem
      2361. +
    2361. With Lasso Regression
      2362. +
    2362. Another Example, now with a polynomial fit
      2363. +
    2363. To think about, first part
      2364. +
    2364. More thinking
      2365. +
    2365. Still thinking
      2366. +
    2366. What does centering (subtracting the mean values) mean mathematically?
      2367. +
    2367. Further Manipulations
      2368. +
    2368. Wrapping it up
      2369. +
    2369. Linear Regression code, Intercept handling first
      2370. +
    2370. Code Examples
      2371. +
    2371. Taking out the mean
      2372. +
    2372. Friday September 9
      2373. +
    2373. Linking the regression analysis with a statistical interpretation
      2374. +
    2374. Assumptions made
      2375. +
    2375. Expectation value and variance
      2376. +
    2376. Expectation value and variance for \( \boldsymbol{\beta} \)
      2377. +
    2377. Deriving OLS from a probability distribution
      2378. +
    2378. Independent and Identically Distrubuted (iid)
      2379. +
    2379. Maximum Likelihood Estimation (MLE)
      2380. +
    2380. A new Cost Function
      2381. +
    2381. More basic Statistics and Bayes' theorem
      2382. +
    2382. Marginal Probability
      2383. +
    2383. Conditional Probability
      2384. +
    2384. Bayes' Theorem
      2385. +
    2385. Interpretations of Bayes' Theorem
      2386. +
    2386. Example of Usage of Bayes' theorem
      2387. +
    2387. Doing it correctly
      2388. +
    2388. Bayes' Theorem and Ridge and Lasso Regression
      2389. +
    2389. Test Function for what happens with OLS, Ridge and Lasso
      2390. +
    2390. Invoking Bayes' theorem
      2391. +
    2391. Ridge and Bayes
      2392. +
    2392. Lasso and Bayes
      2393. +
    2393. Exercise 1: mean values and variances in linear regression
      2394. +
    2394. Exercise 2: Adding Ridge and Lasso Regression
      2395. @@ -297,26 +295,20 @@ MathJax.Hub.Config({

       

       

       

      -

      Lasso Regression

      +

      Yet another Example

      + +

      Let us assume we have a data set with outputs/targets given by the vector

      -

      For Lasso regression our cost function is

      $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, +\boldsymbol{y}=\begin{bmatrix}4 \\ 2 \\3\end{bmatrix}, $$ -

      and minimizing we have that

      +

      and our inputs as a \( 3\times 2 \) design matrix

      $$ --2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, $$ -

      which leads to

      -$$ -\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ - y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ - 0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. -$$ - -

      Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).

      +

      meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.

      @@ -343,7 +335,7 @@ $$

    2395. 29
    2396. 30
    2397. ...
      2398. -
    2398. 60
      2399. +
    2399. 59
    2400. »
      2401. diff --git a/doc/pub/week36/html/._week36-bs021.html b/doc/pub/week36/html/._week36-bs021.html index 2d2c24864..ce416353a 100644 --- a/doc/pub/week36/html/._week36-bs021.html +++ b/doc/pub/week36/html/._week36-bs021.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    2401. Plans for week 36
      2402. -
    2402. Thursday September 8
      2403. -
    2403. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2404. -
    2404. Linear Regression and the SVD
      2405. -
    2405. What does it mean?
      2406. -
    2406. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2407. -
    2407. Code for SVD and Inversion of Matrices
      2408. -
    2408. Inverse of Rectangular Matrix
      2409. -
    2409. Ridge and LASSO Regression
      2410. -
    2410. From OLS to Ridge and Lasso
      2411. -
    2411. Deriving the Ridge Regression Equations
      2412. -
    2412. Note on Scikit-Learn
      2413. -
    2413. Comparison with OLS
      2414. -
    2414. SVD analysis
      2415. -
    2415. Interpreting the Ridge results
      2416. -
    2416. More interpretations
      2417. -
    2417. Deriving the Lasso Regression Equations
      2418. -
    2418. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2419. -
    2419. Ridge Regression
      2420. -
    2420. Lasso Regression
      2421. -
    2421. Yet another Example
      2422. -
    2422. The OLS case
      2423. -
    2423. The Ridge case
      2424. -
    2424. Writing the Cost Function
      2425. -
    2425. Lasso case
      2426. -
    2426. The first Case
      2427. -
    2427. Simple code for solving the above problem
      2428. -
    2428. With Lasso Regression
      2429. -
    2429. Another Example, now with a polynomial fit
      2430. -
    2430. To think about, first part
      2431. -
    2431. More thinking
      2432. -
    2432. Still thinking
      2433. -
    2433. What does centering (subtracting the mean values) mean mathematically?
      2434. -
    2434. Further Manipulations
      2435. -
    2435. Wrapping it up
      2436. -
    2436. Linear Regression code, Intercept handling first
      2437. -
    2437. Code Examples
      2438. -
    2438. Taking out the mean
      2439. -
    2439. Friday September 9
      2440. -
    2440. Linking the regression analysis with a statistical interpretation
      2441. -
    2441. Assumptions made
      2442. -
    2442. Expectation value and variance
      2443. -
    2443. Expectation value and variance for \( \boldsymbol{\beta} \)
      2444. -
    2444. Deriving OLS from a probability distribution
      2445. -
    2445. Independent and Identically Distrubuted (iid)
      2446. -
    2446. Maximum Likelihood Estimation (MLE)
      2447. -
    2447. A new Cost Function
      2448. -
    2448. More basic Statistics and Bayes' theorem
      2449. -
    2449. Marginal Probability
      2450. -
    2450. Conditional Probability
      2451. -
    2451. Bayes' Theorem
      2452. -
    2452. Interpretations of Bayes' Theorem
      2453. -
    2453. Example of Usage of Bayes' theorem
      2454. -
    2454. Doing it correctly
      2455. -
    2455. Bayes' Theorem and Ridge and Lasso Regression
      2456. -
    2456. Test Function for what happens with OLS, Ridge and Lasso
      2457. -
    2457. Invoking Bayes' theorem
      2458. -
    2458. Ridge and Bayes
      2459. -
    2459. Lasso and Bayes
      2460. -
    2460. Exercise 1: mean values and variances in linear regression
      2461. -
    2461. Exercise 2: Adding Ridge and Lasso Regression
      2462. +
    2462. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2463. +
    2463. Linear Regression and the SVD
      2464. +
    2464. What does it mean?
      2465. +
    2465. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2466. +
    2466. Code for SVD and Inversion of Matrices
      2467. +
    2467. Inverse of Rectangular Matrix
      2468. +
    2468. Ridge and LASSO Regression
      2469. +
    2469. From OLS to Ridge and Lasso
      2470. +
    2470. Deriving the Ridge Regression Equations
      2471. +
    2471. Note on Scikit-Learn
      2472. +
    2472. Comparison with OLS
      2473. +
    2473. SVD analysis
      2474. +
    2474. Interpreting the Ridge results
      2475. +
    2475. More interpretations
      2476. +
    2476. Deriving the Lasso Regression Equations
      2477. +
    2477. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2478. +
    2478. Ridge Regression
      2479. +
    2479. Lasso Regression
      2480. +
    2480. Yet another Example
      2481. +
    2481. The OLS case
      2482. +
    2482. The Ridge case
      2483. +
    2483. Writing the Cost Function
      2484. +
    2484. Lasso case
      2485. +
    2485. The first Case
      2486. +
    2486. Simple code for solving the above problem
      2487. +
    2487. With Lasso Regression
      2488. +
    2488. Another Example, now with a polynomial fit
      2489. +
    2489. To think about, first part
      2490. +
    2490. More thinking
      2491. +
    2491. Still thinking
      2492. +
    2492. What does centering (subtracting the mean values) mean mathematically?
      2493. +
    2493. Further Manipulations
      2494. +
    2494. Wrapping it up
      2495. +
    2495. Linear Regression code, Intercept handling first
      2496. +
    2496. Code Examples
      2497. +
    2497. Taking out the mean
      2498. +
    2498. Friday September 9
      2499. +
    2499. Linking the regression analysis with a statistical interpretation
      2500. +
    2500. Assumptions made
      2501. +
    2501. Expectation value and variance
      2502. +
    2502. Expectation value and variance for \( \boldsymbol{\beta} \)
      2503. +
    2503. Deriving OLS from a probability distribution
      2504. +
    2504. Independent and Identically Distrubuted (iid)
      2505. +
    2505. Maximum Likelihood Estimation (MLE)
      2506. +
    2506. A new Cost Function
      2507. +
    2507. More basic Statistics and Bayes' theorem
      2508. +
    2508. Marginal Probability
      2509. +
    2509. Conditional Probability
      2510. +
    2510. Bayes' Theorem
      2511. +
    2511. Interpretations of Bayes' Theorem
      2512. +
    2512. Example of Usage of Bayes' theorem
      2513. +
    2513. Doing it correctly
      2514. +
    2514. Bayes' Theorem and Ridge and Lasso Regression
      2515. +
    2515. Test Function for what happens with OLS, Ridge and Lasso
      2516. +
    2516. Invoking Bayes' theorem
      2517. +
    2517. Ridge and Bayes
      2518. +
    2518. Lasso and Bayes
      2519. +
    2519. Exercise 1: mean values and variances in linear regression
      2520. +
    2520. Exercise 2: Adding Ridge and Lasso Regression
      2521. @@ -297,20 +295,21 @@ MathJax.Hub.Config({

       

       

       

      -

      Yet another Example

      +

      The OLS case

      -

      Let us assume we have a data set with outputs/targets given by the vector

      +

      For ordinary least squares (OLS) we know that the optimal solution is

      $$ -\boldsymbol{y}=\begin{bmatrix}4 \\ 2 \\3\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

      and our inputs as a \( 3\times 2 \) design matrix

      +

      Inserting the above values we obtain that

      + $$ -\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, $$ -

      meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.

      +

      The code which implements this simpler case is presented after the discussion of Ridge and Lasso.

      @@ -337,7 +336,7 @@ $$

    2521. 30
    2522. 31
    2523. ...
      2524. -
    2524. 60
      2525. +
    2525. 59
    2526. »
      2527. diff --git a/doc/pub/week36/html/._week36-bs022.html b/doc/pub/week36/html/._week36-bs022.html index 0136ae958..f074e3484 100644 --- a/doc/pub/week36/html/._week36-bs022.html +++ b/doc/pub/week36/html/._week36-bs022.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    2527. Plans for week 36
      2528. -
    2528. Thursday September 8
      2529. -
    2529. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2530. -
    2530. Linear Regression and the SVD
      2531. -
    2531. What does it mean?
      2532. -
    2532. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2533. -
    2533. Code for SVD and Inversion of Matrices
      2534. -
    2534. Inverse of Rectangular Matrix
      2535. -
    2535. Ridge and LASSO Regression
      2536. -
    2536. From OLS to Ridge and Lasso
      2537. -
    2537. Deriving the Ridge Regression Equations
      2538. -
    2538. Note on Scikit-Learn
      2539. -
    2539. Comparison with OLS
      2540. -
    2540. SVD analysis
      2541. -
    2541. Interpreting the Ridge results
      2542. -
    2542. More interpretations
      2543. -
    2543. Deriving the Lasso Regression Equations
      2544. -
    2544. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2545. -
    2545. Ridge Regression
      2546. -
    2546. Lasso Regression
      2547. -
    2547. Yet another Example
      2548. -
    2548. The OLS case
      2549. -
    2549. The Ridge case
      2550. -
    2550. Writing the Cost Function
      2551. -
    2551. Lasso case
      2552. -
    2552. The first Case
      2553. -
    2553. Simple code for solving the above problem
      2554. -
    2554. With Lasso Regression
      2555. -
    2555. Another Example, now with a polynomial fit
      2556. -
    2556. To think about, first part
      2557. -
    2557. More thinking
      2558. -
    2558. Still thinking
      2559. -
    2559. What does centering (subtracting the mean values) mean mathematically?
      2560. -
    2560. Further Manipulations
      2561. -
    2561. Wrapping it up
      2562. -
    2562. Linear Regression code, Intercept handling first
      2563. -
    2563. Code Examples
      2564. -
    2564. Taking out the mean
      2565. -
    2565. Friday September 9
      2566. -
    2566. Linking the regression analysis with a statistical interpretation
      2567. -
    2567. Assumptions made
      2568. -
    2568. Expectation value and variance
      2569. -
    2569. Expectation value and variance for \( \boldsymbol{\beta} \)
      2570. -
    2570. Deriving OLS from a probability distribution
      2571. -
    2571. Independent and Identically Distrubuted (iid)
      2572. -
    2572. Maximum Likelihood Estimation (MLE)
      2573. -
    2573. A new Cost Function
      2574. -
    2574. More basic Statistics and Bayes' theorem
      2575. -
    2575. Marginal Probability
      2576. -
    2576. Conditional Probability
      2577. -
    2577. Bayes' Theorem
      2578. -
    2578. Interpretations of Bayes' Theorem
      2579. -
    2579. Example of Usage of Bayes' theorem
      2580. -
    2580. Doing it correctly
      2581. -
    2581. Bayes' Theorem and Ridge and Lasso Regression
      2582. -
    2582. Test Function for what happens with OLS, Ridge and Lasso
      2583. -
    2583. Invoking Bayes' theorem
      2584. -
    2584. Ridge and Bayes
      2585. -
    2585. Lasso and Bayes
      2586. -
    2586. Exercise 1: mean values and variances in linear regression
      2587. -
    2587. Exercise 2: Adding Ridge and Lasso Regression
      2588. +
    2588. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2589. +
    2589. Linear Regression and the SVD
      2590. +
    2590. What does it mean?
      2591. +
    2591. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2592. +
    2592. Code for SVD and Inversion of Matrices
      2593. +
    2593. Inverse of Rectangular Matrix
      2594. +
    2594. Ridge and LASSO Regression
      2595. +
    2595. From OLS to Ridge and Lasso
      2596. +
    2596. Deriving the Ridge Regression Equations
      2597. +
    2597. Note on Scikit-Learn
      2598. +
    2598. Comparison with OLS
      2599. +
    2599. SVD analysis
      2600. +
    2600. Interpreting the Ridge results
      2601. +
    2601. More interpretations
      2602. +
    2602. Deriving the Lasso Regression Equations
      2603. +
    2603. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2604. +
    2604. Ridge Regression
      2605. +
    2605. Lasso Regression
      2606. +
    2606. Yet another Example
      2607. +
    2607. The OLS case
      2608. +
    2608. The Ridge case
      2609. +
    2609. Writing the Cost Function
      2610. +
    2610. Lasso case
      2611. +
    2611. The first Case
      2612. +
    2612. Simple code for solving the above problem
      2613. +
    2613. With Lasso Regression
      2614. +
    2614. Another Example, now with a polynomial fit
      2615. +
    2615. To think about, first part
      2616. +
    2616. More thinking
      2617. +
    2617. Still thinking
      2618. +
    2618. What does centering (subtracting the mean values) mean mathematically?
      2619. +
    2619. Further Manipulations
      2620. +
    2620. Wrapping it up
      2621. +
    2621. Linear Regression code, Intercept handling first
      2622. +
    2622. Code Examples
      2623. +
    2623. Taking out the mean
      2624. +
    2624. Friday September 9
      2625. +
    2625. Linking the regression analysis with a statistical interpretation
      2626. +
    2626. Assumptions made
      2627. +
    2627. Expectation value and variance
      2628. +
    2628. Expectation value and variance for \( \boldsymbol{\beta} \)
      2629. +
    2629. Deriving OLS from a probability distribution
      2630. +
    2630. Independent and Identically Distrubuted (iid)
      2631. +
    2631. Maximum Likelihood Estimation (MLE)
      2632. +
    2632. A new Cost Function
      2633. +
    2633. More basic Statistics and Bayes' theorem
      2634. +
    2634. Marginal Probability
      2635. +
    2635. Conditional Probability
      2636. +
    2636. Bayes' Theorem
      2637. +
    2637. Interpretations of Bayes' Theorem
      2638. +
    2638. Example of Usage of Bayes' theorem
      2639. +
    2639. Doing it correctly
      2640. +
    2640. Bayes' Theorem and Ridge and Lasso Regression
      2641. +
    2641. Test Function for what happens with OLS, Ridge and Lasso
      2642. +
    2642. Invoking Bayes' theorem
      2643. +
    2643. Ridge and Bayes
      2644. +
    2644. Lasso and Bayes
      2645. +
    2645. Exercise 1: mean values and variances in linear regression
      2646. +
    2646. Exercise 2: Adding Ridge and Lasso Regression
      2647. @@ -297,21 +295,25 @@ MathJax.Hub.Config({

       

       

       

      -

      The OLS case

      +

      The Ridge case

      -

      For ordinary least squares (OLS) we know that the optimal solution is

      +

      For Ridge regression we have

      $$ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$

      Inserting the above values we obtain that

      $$ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, $$ -

      The code which implements this simpler case is presented after the discussion of Ridge and Lasso.

      +

      There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \). +

      + +

      To see this, let us write the cost function for Ridge regression.

      @@ -338,7 +340,7 @@ $$

    2647. 31
    2648. 32
    2649. ...
      2650. -
    2650. 60
      2651. +
    2651. 59
    2652. »
      2653. diff --git a/doc/pub/week36/html/._week36-bs023.html b/doc/pub/week36/html/._week36-bs023.html index ebd843e94..3c8b0d906 100644 --- a/doc/pub/week36/html/._week36-bs023.html +++ b/doc/pub/week36/html/._week36-bs023.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    2653. Plans for week 36
      2654. -
    2654. Thursday September 8
      2655. -
    2655. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2656. -
    2656. Linear Regression and the SVD
      2657. -
    2657. What does it mean?
      2658. -
    2658. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2659. -
    2659. Code for SVD and Inversion of Matrices
      2660. -
    2660. Inverse of Rectangular Matrix
      2661. -
    2661. Ridge and LASSO Regression
      2662. -
    2662. From OLS to Ridge and Lasso
      2663. -
    2663. Deriving the Ridge Regression Equations
      2664. -
    2664. Note on Scikit-Learn
      2665. -
    2665. Comparison with OLS
      2666. -
    2666. SVD analysis
      2667. -
    2667. Interpreting the Ridge results
      2668. -
    2668. More interpretations
      2669. -
    2669. Deriving the Lasso Regression Equations
      2670. -
    2670. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2671. -
    2671. Ridge Regression
      2672. -
    2672. Lasso Regression
      2673. -
    2673. Yet another Example
      2674. -
    2674. The OLS case
      2675. -
    2675. The Ridge case
      2676. -
    2676. Writing the Cost Function
      2677. -
    2677. Lasso case
      2678. -
    2678. The first Case
      2679. -
    2679. Simple code for solving the above problem
      2680. -
    2680. With Lasso Regression
      2681. -
    2681. Another Example, now with a polynomial fit
      2682. -
    2682. To think about, first part
      2683. -
    2683. More thinking
      2684. -
    2684. Still thinking
      2685. -
    2685. What does centering (subtracting the mean values) mean mathematically?
      2686. -
    2686. Further Manipulations
      2687. -
    2687. Wrapping it up
      2688. -
    2688. Linear Regression code, Intercept handling first
      2689. -
    2689. Code Examples
      2690. -
    2690. Taking out the mean
      2691. -
    2691. Friday September 9
      2692. -
    2692. Linking the regression analysis with a statistical interpretation
      2693. -
    2693. Assumptions made
      2694. -
    2694. Expectation value and variance
      2695. -
    2695. Expectation value and variance for \( \boldsymbol{\beta} \)
      2696. -
    2696. Deriving OLS from a probability distribution
      2697. -
    2697. Independent and Identically Distrubuted (iid)
      2698. -
    2698. Maximum Likelihood Estimation (MLE)
      2699. -
    2699. A new Cost Function
      2700. -
    2700. More basic Statistics and Bayes' theorem
      2701. -
    2701. Marginal Probability
      2702. -
    2702. Conditional Probability
      2703. -
    2703. Bayes' Theorem
      2704. -
    2704. Interpretations of Bayes' Theorem
      2705. -
    2705. Example of Usage of Bayes' theorem
      2706. -
    2706. Doing it correctly
      2707. -
    2707. Bayes' Theorem and Ridge and Lasso Regression
      2708. -
    2708. Test Function for what happens with OLS, Ridge and Lasso
      2709. -
    2709. Invoking Bayes' theorem
      2710. -
    2710. Ridge and Bayes
      2711. -
    2711. Lasso and Bayes
      2712. -
    2712. Exercise 1: mean values and variances in linear regression
      2713. -
    2713. Exercise 2: Adding Ridge and Lasso Regression
      2714. +
    2714. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2715. +
    2715. Linear Regression and the SVD
      2716. +
    2716. What does it mean?
      2717. +
    2717. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2718. +
    2718. Code for SVD and Inversion of Matrices
      2719. +
    2719. Inverse of Rectangular Matrix
      2720. +
    2720. Ridge and LASSO Regression
      2721. +
    2721. From OLS to Ridge and Lasso
      2722. +
    2722. Deriving the Ridge Regression Equations
      2723. +
    2723. Note on Scikit-Learn
      2724. +
    2724. Comparison with OLS
      2725. +
    2725. SVD analysis
      2726. +
    2726. Interpreting the Ridge results
      2727. +
    2727. More interpretations
      2728. +
    2728. Deriving the Lasso Regression Equations
      2729. +
    2729. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2730. +
    2730. Ridge Regression
      2731. +
    2731. Lasso Regression
      2732. +
    2732. Yet another Example
      2733. +
    2733. The OLS case
      2734. +
    2734. The Ridge case
      2735. +
    2735. Writing the Cost Function
      2736. +
    2736. Lasso case
      2737. +
    2737. The first Case
      2738. +
    2738. Simple code for solving the above problem
      2739. +
    2739. With Lasso Regression
      2740. +
    2740. Another Example, now with a polynomial fit
      2741. +
    2741. To think about, first part
      2742. +
    2742. More thinking
      2743. +
    2743. Still thinking
      2744. +
    2744. What does centering (subtracting the mean values) mean mathematically?
      2745. +
    2745. Further Manipulations
      2746. +
    2746. Wrapping it up
      2747. +
    2747. Linear Regression code, Intercept handling first
      2748. +
    2748. Code Examples
      2749. +
    2749. Taking out the mean
      2750. +
    2750. Friday September 9
      2751. +
    2751. Linking the regression analysis with a statistical interpretation
      2752. +
    2752. Assumptions made
      2753. +
    2753. Expectation value and variance
      2754. +
    2754. Expectation value and variance for \( \boldsymbol{\beta} \)
      2755. +
    2755. Deriving OLS from a probability distribution
      2756. +
    2756. Independent and Identically Distrubuted (iid)
      2757. +
    2757. Maximum Likelihood Estimation (MLE)
      2758. +
    2758. A new Cost Function
      2759. +
    2759. More basic Statistics and Bayes' theorem
      2760. +
    2760. Marginal Probability
      2761. +
    2761. Conditional Probability
      2762. +
    2762. Bayes' Theorem
      2763. +
    2763. Interpretations of Bayes' Theorem
      2764. +
    2764. Example of Usage of Bayes' theorem
      2765. +
    2765. Doing it correctly
      2766. +
    2766. Bayes' Theorem and Ridge and Lasso Regression
      2767. +
    2767. Test Function for what happens with OLS, Ridge and Lasso
      2768. +
    2768. Invoking Bayes' theorem
      2769. +
    2769. Ridge and Bayes
      2770. +
    2770. Lasso and Bayes
      2771. +
    2771. Exercise 1: mean values and variances in linear regression
      2772. +
    2772. Exercise 2: Adding Ridge and Lasso Regression
      2773. @@ -297,25 +295,34 @@ MathJax.Hub.Config({

       

       

       

      -

      The Ridge case

      - -

      For Ridge regression we have

      +

      Writing the Cost Function

      +

      We define the MSE without the \( 1/n \) factor and have then, using that

      $$ -\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, $$ -

      Inserting the above values we obtain that

      $$ -\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), $$ -

      There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). -Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \). -

      +

      and taking the derivative with respect to \( \beta_0 \) we get

      +$$ +\beta_0=\frac{8}{4+\lambda}, +$$ -

      To see this, let us write the cost function for Ridge regression.

      +

      and for \( \beta_1 \) we obtain

      +$$ +\beta_1=\frac{2}{1+\lambda}, +$$ + +

      Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving

      +$$ +\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, +$$ + +

      which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).

      @@ -342,7 +349,7 @@ Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. Thi

    2773. 32
    2774. 33
    2775. ...
      2776. -
    2776. 60
      2777. +
    2777. 59
    2778. »
      2779. diff --git a/doc/pub/week36/html/._week36-bs024.html b/doc/pub/week36/html/._week36-bs024.html index d8b66110d..312eb3626 100644 --- a/doc/pub/week36/html/._week36-bs024.html +++ b/doc/pub/week36/html/._week36-bs024.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    2779. Plans for week 36
      2780. -
    2780. Thursday September 8
      2781. -
    2781. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2782. -
    2782. Linear Regression and the SVD
      2783. -
    2783. What does it mean?
      2784. -
    2784. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2785. -
    2785. Code for SVD and Inversion of Matrices
      2786. -
    2786. Inverse of Rectangular Matrix
      2787. -
    2787. Ridge and LASSO Regression
      2788. -
    2788. From OLS to Ridge and Lasso
      2789. -
    2789. Deriving the Ridge Regression Equations
      2790. -
    2790. Note on Scikit-Learn
      2791. -
    2791. Comparison with OLS
      2792. -
    2792. SVD analysis
      2793. -
    2793. Interpreting the Ridge results
      2794. -
    2794. More interpretations
      2795. -
    2795. Deriving the Lasso Regression Equations
      2796. -
    2796. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2797. -
    2797. Ridge Regression
      2798. -
    2798. Lasso Regression
      2799. -
    2799. Yet another Example
      2800. -
    2800. The OLS case
      2801. -
    2801. The Ridge case
      2802. -
    2802. Writing the Cost Function
      2803. -
    2803. Lasso case
      2804. -
    2804. The first Case
      2805. -
    2805. Simple code for solving the above problem
      2806. -
    2806. With Lasso Regression
      2807. -
    2807. Another Example, now with a polynomial fit
      2808. -
    2808. To think about, first part
      2809. -
    2809. More thinking
      2810. -
    2810. Still thinking
      2811. -
    2811. What does centering (subtracting the mean values) mean mathematically?
      2812. -
    2812. Further Manipulations
      2813. -
    2813. Wrapping it up
      2814. -
    2814. Linear Regression code, Intercept handling first
      2815. -
    2815. Code Examples
      2816. -
    2816. Taking out the mean
      2817. -
    2817. Friday September 9
      2818. -
    2818. Linking the regression analysis with a statistical interpretation
      2819. -
    2819. Assumptions made
      2820. -
    2820. Expectation value and variance
      2821. -
    2821. Expectation value and variance for \( \boldsymbol{\beta} \)
      2822. -
    2822. Deriving OLS from a probability distribution
      2823. -
    2823. Independent and Identically Distrubuted (iid)
      2824. -
    2824. Maximum Likelihood Estimation (MLE)
      2825. -
    2825. A new Cost Function
      2826. -
    2826. More basic Statistics and Bayes' theorem
      2827. -
    2827. Marginal Probability
      2828. -
    2828. Conditional Probability
      2829. -
    2829. Bayes' Theorem
      2830. -
    2830. Interpretations of Bayes' Theorem
      2831. -
    2831. Example of Usage of Bayes' theorem
      2832. -
    2832. Doing it correctly
      2833. -
    2833. Bayes' Theorem and Ridge and Lasso Regression
      2834. -
    2834. Test Function for what happens with OLS, Ridge and Lasso
      2835. -
    2835. Invoking Bayes' theorem
      2836. -
    2836. Ridge and Bayes
      2837. -
    2837. Lasso and Bayes
      2838. -
    2838. Exercise 1: mean values and variances in linear regression
      2839. -
    2839. Exercise 2: Adding Ridge and Lasso Regression
      2840. +
    2840. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2841. +
    2841. Linear Regression and the SVD
      2842. +
    2842. What does it mean?
      2843. +
    2843. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2844. +
    2844. Code for SVD and Inversion of Matrices
      2845. +
    2845. Inverse of Rectangular Matrix
      2846. +
    2846. Ridge and LASSO Regression
      2847. +
    2847. From OLS to Ridge and Lasso
      2848. +
    2848. Deriving the Ridge Regression Equations
      2849. +
    2849. Note on Scikit-Learn
      2850. +
    2850. Comparison with OLS
      2851. +
    2851. SVD analysis
      2852. +
    2852. Interpreting the Ridge results
      2853. +
    2853. More interpretations
      2854. +
    2854. Deriving the Lasso Regression Equations
      2855. +
    2855. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2856. +
    2856. Ridge Regression
      2857. +
    2857. Lasso Regression
      2858. +
    2858. Yet another Example
      2859. +
    2859. The OLS case
      2860. +
    2860. The Ridge case
      2861. +
    2861. Writing the Cost Function
      2862. +
    2862. Lasso case
      2863. +
    2863. The first Case
      2864. +
    2864. Simple code for solving the above problem
      2865. +
    2865. With Lasso Regression
      2866. +
    2866. Another Example, now with a polynomial fit
      2867. +
    2867. To think about, first part
      2868. +
    2868. More thinking
      2869. +
    2869. Still thinking
      2870. +
    2870. What does centering (subtracting the mean values) mean mathematically?
      2871. +
    2871. Further Manipulations
      2872. +
    2872. Wrapping it up
      2873. +
    2873. Linear Regression code, Intercept handling first
      2874. +
    2874. Code Examples
      2875. +
    2875. Taking out the mean
      2876. +
    2876. Friday September 9
      2877. +
    2877. Linking the regression analysis with a statistical interpretation
      2878. +
    2878. Assumptions made
      2879. +
    2879. Expectation value and variance
      2880. +
    2880. Expectation value and variance for \( \boldsymbol{\beta} \)
      2881. +
    2881. Deriving OLS from a probability distribution
      2882. +
    2882. Independent and Identically Distrubuted (iid)
      2883. +
    2883. Maximum Likelihood Estimation (MLE)
      2884. +
    2884. A new Cost Function
      2885. +
    2885. More basic Statistics and Bayes' theorem
      2886. +
    2886. Marginal Probability
      2887. +
    2887. Conditional Probability
      2888. +
    2888. Bayes' Theorem
      2889. +
    2889. Interpretations of Bayes' Theorem
      2890. +
    2890. Example of Usage of Bayes' theorem
      2891. +
    2891. Doing it correctly
      2892. +
    2892. Bayes' Theorem and Ridge and Lasso Regression
      2893. +
    2893. Test Function for what happens with OLS, Ridge and Lasso
      2894. +
    2894. Invoking Bayes' theorem
      2895. +
    2895. Ridge and Bayes
      2896. +
    2896. Lasso and Bayes
      2897. +
    2897. Exercise 1: mean values and variances in linear regression
      2898. +
    2898. Exercise 2: Adding Ridge and Lasso Regression
      2899. @@ -297,35 +295,32 @@ MathJax.Hub.Config({

       

       

       

      -

      Writing the Cost Function

      +

      Lasso case

      -

      We define the MSE without the \( 1/n \) factor and have then, using that

      +

      For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) +and \( \beta_1 \). This gives us the following derivatives of the cost function +

      $$ -\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, +C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), $$ $$ -C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, $$ -

      and taking the derivative with respect to \( \beta_0 \) we get

      +

      and

      $$ -\beta_0=\frac{8}{4+\lambda}, +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. $$ -

      and for \( \beta_1 \) we obtain

      -$$ -\beta_1=\frac{2}{1+\lambda}, -$$ - -

      Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving

      -$$ -\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, -$$ - -

      which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).

      - +

      We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely

      +
        +
      1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
        2. +
      2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
        3. +
      3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
        4. +
      4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \).
        5. +

      diff --git a/doc/pub/week36/html/._week36-bs025.html b/doc/pub/week36/html/._week36-bs025.html index ba60331b2..69a2195ae 100644 --- a/doc/pub/week36/html/._week36-bs025.html +++ b/doc/pub/week36/html/._week36-bs025.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    2899. Plans for week 36
      2900. -
    2900. Thursday September 8
      2901. -
    2901. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2902. -
    2902. Linear Regression and the SVD
      2903. -
    2903. What does it mean?
      2904. -
    2904. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2905. -
    2905. Code for SVD and Inversion of Matrices
      2906. -
    2906. Inverse of Rectangular Matrix
      2907. -
    2907. Ridge and LASSO Regression
      2908. -
    2908. From OLS to Ridge and Lasso
      2909. -
    2909. Deriving the Ridge Regression Equations
      2910. -
    2910. Note on Scikit-Learn
      2911. -
    2911. Comparison with OLS
      2912. -
    2912. SVD analysis
      2913. -
    2913. Interpreting the Ridge results
      2914. -
    2914. More interpretations
      2915. -
    2915. Deriving the Lasso Regression Equations
      2916. -
    2916. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2917. -
    2917. Ridge Regression
      2918. -
    2918. Lasso Regression
      2919. -
    2919. Yet another Example
      2920. -
    2920. The OLS case
      2921. -
    2921. The Ridge case
      2922. -
    2922. Writing the Cost Function
      2923. -
    2923. Lasso case
      2924. -
    2924. The first Case
      2925. -
    2925. Simple code for solving the above problem
      2926. -
    2926. With Lasso Regression
      2927. -
    2927. Another Example, now with a polynomial fit
      2928. -
    2928. To think about, first part
      2929. -
    2929. More thinking
      2930. -
    2930. Still thinking
      2931. -
    2931. What does centering (subtracting the mean values) mean mathematically?
      2932. -
    2932. Further Manipulations
      2933. -
    2933. Wrapping it up
      2934. -
    2934. Linear Regression code, Intercept handling first
      2935. -
    2935. Code Examples
      2936. -
    2936. Taking out the mean
      2937. -
    2937. Friday September 9
      2938. -
    2938. Linking the regression analysis with a statistical interpretation
      2939. -
    2939. Assumptions made
      2940. -
    2940. Expectation value and variance
      2941. -
    2941. Expectation value and variance for \( \boldsymbol{\beta} \)
      2942. -
    2942. Deriving OLS from a probability distribution
      2943. -
    2943. Independent and Identically Distrubuted (iid)
      2944. -
    2944. Maximum Likelihood Estimation (MLE)
      2945. -
    2945. A new Cost Function
      2946. -
    2946. More basic Statistics and Bayes' theorem
      2947. -
    2947. Marginal Probability
      2948. -
    2948. Conditional Probability
      2949. -
    2949. Bayes' Theorem
      2950. -
    2950. Interpretations of Bayes' Theorem
      2951. -
    2951. Example of Usage of Bayes' theorem
      2952. -
    2952. Doing it correctly
      2953. -
    2953. Bayes' Theorem and Ridge and Lasso Regression
      2954. -
    2954. Test Function for what happens with OLS, Ridge and Lasso
      2955. -
    2955. Invoking Bayes' theorem
      2956. -
    2956. Ridge and Bayes
      2957. -
    2957. Lasso and Bayes
      2958. -
    2958. Exercise 1: mean values and variances in linear regression
      2959. -
    2959. Exercise 2: Adding Ridge and Lasso Regression
      2960. +
    2960. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      2961. +
    2961. Linear Regression and the SVD
      2962. +
    2962. What does it mean?
      2963. +
    2963. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      2964. +
    2964. Code for SVD and Inversion of Matrices
      2965. +
    2965. Inverse of Rectangular Matrix
      2966. +
    2966. Ridge and LASSO Regression
      2967. +
    2967. From OLS to Ridge and Lasso
      2968. +
    2968. Deriving the Ridge Regression Equations
      2969. +
    2969. Note on Scikit-Learn
      2970. +
    2970. Comparison with OLS
      2971. +
    2971. SVD analysis
      2972. +
    2972. Interpreting the Ridge results
      2973. +
    2973. More interpretations
      2974. +
    2974. Deriving the Lasso Regression Equations
      2975. +
    2975. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      2976. +
    2976. Ridge Regression
      2977. +
    2977. Lasso Regression
      2978. +
    2978. Yet another Example
      2979. +
    2979. The OLS case
      2980. +
    2980. The Ridge case
      2981. +
    2981. Writing the Cost Function
      2982. +
    2982. Lasso case
      2983. +
    2983. The first Case
      2984. +
    2984. Simple code for solving the above problem
      2985. +
    2985. With Lasso Regression
      2986. +
    2986. Another Example, now with a polynomial fit
      2987. +
    2987. To think about, first part
      2988. +
    2988. More thinking
      2989. +
    2989. Still thinking
      2990. +
    2990. What does centering (subtracting the mean values) mean mathematically?
      2991. +
    2991. Further Manipulations
      2992. +
    2992. Wrapping it up
      2993. +
    2993. Linear Regression code, Intercept handling first
      2994. +
    2994. Code Examples
      2995. +
    2995. Taking out the mean
      2996. +
    2996. Friday September 9
      2997. +
    2997. Linking the regression analysis with a statistical interpretation
      2998. +
    2998. Assumptions made
      2999. +
    2999. Expectation value and variance
      3000. +
    3000. Expectation value and variance for \( \boldsymbol{\beta} \)
      3001. +
    3001. Deriving OLS from a probability distribution
      3002. +
    3002. Independent and Identically Distrubuted (iid)
      3003. +
    3003. Maximum Likelihood Estimation (MLE)
      3004. +
    3004. A new Cost Function
      3005. +
    3005. More basic Statistics and Bayes' theorem
      3006. +
    3006. Marginal Probability
      3007. +
    3007. Conditional Probability
      3008. +
    3008. Bayes' Theorem
      3009. +
    3009. Interpretations of Bayes' Theorem
      3010. +
    3010. Example of Usage of Bayes' theorem
      3011. +
    3011. Doing it correctly
      3012. +
    3012. Bayes' Theorem and Ridge and Lasso Regression
      3013. +
    3013. Test Function for what happens with OLS, Ridge and Lasso
      3014. +
    3014. Invoking Bayes' theorem
      3015. +
    3015. Ridge and Bayes
      3016. +
    3016. Lasso and Bayes
      3017. +
    3017. Exercise 1: mean values and variances in linear regression
      3018. +
    3018. Exercise 2: Adding Ridge and Lasso Regression
      3019. @@ -297,32 +295,31 @@ MathJax.Hub.Config({

       

       

       

      -

      Lasso case

      +

      The first Case

      -

      For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) -and \( \beta_1 \). This gives us the following derivatives of the cost function -

      +

      If we consider the first case, we have then

      $$ -C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), -$$ - - -$$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, +-4(4-2\beta_0)+\lambda=0, $$

      and

      $$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. +-2(2-\beta_1)+\lambda=0. $$ -

      We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely

      -
        -
      1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
        2. -
      2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
        3. -
      3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
        4. -
      4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \).
        5. -
      +

      which yields

      + +$$ +\beta_0=\frac{16+\lambda}{8}, +$$ + +

      and

      +$$ +\beta_1=\frac{4+\lambda}{2}. +$$ + +

      Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you.

      +

      diff --git a/doc/pub/week36/html/._week36-bs026.html b/doc/pub/week36/html/._week36-bs026.html index 06abdeea0..f294e6484 100644 --- a/doc/pub/week36/html/._week36-bs026.html +++ b/doc/pub/week36/html/._week36-bs026.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    3019. Plans for week 36
      3020. -
    3020. Thursday September 8
      3021. -
    3021. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3022. -
    3022. Linear Regression and the SVD
      3023. -
    3023. What does it mean?
      3024. -
    3024. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3025. -
    3025. Code for SVD and Inversion of Matrices
      3026. -
    3026. Inverse of Rectangular Matrix
      3027. -
    3027. Ridge and LASSO Regression
      3028. -
    3028. From OLS to Ridge and Lasso
      3029. -
    3029. Deriving the Ridge Regression Equations
      3030. -
    3030. Note on Scikit-Learn
      3031. -
    3031. Comparison with OLS
      3032. -
    3032. SVD analysis
      3033. -
    3033. Interpreting the Ridge results
      3034. -
    3034. More interpretations
      3035. -
    3035. Deriving the Lasso Regression Equations
      3036. -
    3036. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3037. -
    3037. Ridge Regression
      3038. -
    3038. Lasso Regression
      3039. -
    3039. Yet another Example
      3040. -
    3040. The OLS case
      3041. -
    3041. The Ridge case
      3042. -
    3042. Writing the Cost Function
      3043. -
    3043. Lasso case
      3044. -
    3044. The first Case
      3045. -
    3045. Simple code for solving the above problem
      3046. -
    3046. With Lasso Regression
      3047. -
    3047. Another Example, now with a polynomial fit
      3048. -
    3048. To think about, first part
      3049. -
    3049. More thinking
      3050. -
    3050. Still thinking
      3051. -
    3051. What does centering (subtracting the mean values) mean mathematically?
      3052. -
    3052. Further Manipulations
      3053. -
    3053. Wrapping it up
      3054. -
    3054. Linear Regression code, Intercept handling first
      3055. -
    3055. Code Examples
      3056. -
    3056. Taking out the mean
      3057. -
    3057. Friday September 9
      3058. -
    3058. Linking the regression analysis with a statistical interpretation
      3059. -
    3059. Assumptions made
      3060. -
    3060. Expectation value and variance
      3061. -
    3061. Expectation value and variance for \( \boldsymbol{\beta} \)
      3062. -
    3062. Deriving OLS from a probability distribution
      3063. -
    3063. Independent and Identically Distrubuted (iid)
      3064. -
    3064. Maximum Likelihood Estimation (MLE)
      3065. -
    3065. A new Cost Function
      3066. -
    3066. More basic Statistics and Bayes' theorem
      3067. -
    3067. Marginal Probability
      3068. -
    3068. Conditional Probability
      3069. -
    3069. Bayes' Theorem
      3070. -
    3070. Interpretations of Bayes' Theorem
      3071. -
    3071. Example of Usage of Bayes' theorem
      3072. -
    3072. Doing it correctly
      3073. -
    3073. Bayes' Theorem and Ridge and Lasso Regression
      3074. -
    3074. Test Function for what happens with OLS, Ridge and Lasso
      3075. -
    3075. Invoking Bayes' theorem
      3076. -
    3076. Ridge and Bayes
      3077. -
    3077. Lasso and Bayes
      3078. -
    3078. Exercise 1: mean values and variances in linear regression
      3079. -
    3079. Exercise 2: Adding Ridge and Lasso Regression
      3080. +
    3080. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3081. +
    3081. Linear Regression and the SVD
      3082. +
    3082. What does it mean?
      3083. +
    3083. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3084. +
    3084. Code for SVD and Inversion of Matrices
      3085. +
    3085. Inverse of Rectangular Matrix
      3086. +
    3086. Ridge and LASSO Regression
      3087. +
    3087. From OLS to Ridge and Lasso
      3088. +
    3088. Deriving the Ridge Regression Equations
      3089. +
    3089. Note on Scikit-Learn
      3090. +
    3090. Comparison with OLS
      3091. +
    3091. SVD analysis
      3092. +
    3092. Interpreting the Ridge results
      3093. +
    3093. More interpretations
      3094. +
    3094. Deriving the Lasso Regression Equations
      3095. +
    3095. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3096. +
    3096. Ridge Regression
      3097. +
    3097. Lasso Regression
      3098. +
    3098. Yet another Example
      3099. +
    3099. The OLS case
      3100. +
    3100. The Ridge case
      3101. +
    3101. Writing the Cost Function
      3102. +
    3102. Lasso case
      3103. +
    3103. The first Case
      3104. +
    3104. Simple code for solving the above problem
      3105. +
    3105. With Lasso Regression
      3106. +
    3106. Another Example, now with a polynomial fit
      3107. +
    3107. To think about, first part
      3108. +
    3108. More thinking
      3109. +
    3109. Still thinking
      3110. +
    3110. What does centering (subtracting the mean values) mean mathematically?
      3111. +
    3111. Further Manipulations
      3112. +
    3112. Wrapping it up
      3113. +
    3113. Linear Regression code, Intercept handling first
      3114. +
    3114. Code Examples
      3115. +
    3115. Taking out the mean
      3116. +
    3116. Friday September 9
      3117. +
    3117. Linking the regression analysis with a statistical interpretation
      3118. +
    3118. Assumptions made
      3119. +
    3119. Expectation value and variance
      3120. +
    3120. Expectation value and variance for \( \boldsymbol{\beta} \)
      3121. +
    3121. Deriving OLS from a probability distribution
      3122. +
    3122. Independent and Identically Distrubuted (iid)
      3123. +
    3123. Maximum Likelihood Estimation (MLE)
      3124. +
    3124. A new Cost Function
      3125. +
    3125. More basic Statistics and Bayes' theorem
      3126. +
    3126. Marginal Probability
      3127. +
    3127. Conditional Probability
      3128. +
    3128. Bayes' Theorem
      3129. +
    3129. Interpretations of Bayes' Theorem
      3130. +
    3130. Example of Usage of Bayes' theorem
      3131. +
    3131. Doing it correctly
      3132. +
    3132. Bayes' Theorem and Ridge and Lasso Regression
      3133. +
    3133. Test Function for what happens with OLS, Ridge and Lasso
      3134. +
    3134. Invoking Bayes' theorem
      3135. +
    3135. Ridge and Bayes
      3136. +
    3136. Lasso and Bayes
      3137. +
    3137. Exercise 1: mean values and variances in linear regression
      3138. +
    3138. Exercise 2: Adding Ridge and Lasso Regression
      3139. @@ -297,30 +295,84 @@ MathJax.Hub.Config({

       

       

       

      -

      The first Case

      +

      Simple code for solving the above problem

      -

      If we consider the first case, we have then

      -$$ --4(4-2\beta_0)+\lambda=0, -$$ +

      Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values.

      -

      and

      -$$ --2(2-\beta_1)+\lambda=0. -$$ +

      First we study and compare the OLS and Ridge results. The next code compares all three methods.

      -

      which yields

      -$$ -\beta_0=\frac{16+\lambda}{8}, -$$ + +
      +
      +
      +
      +
      + 
      import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
 
-
      and
      
-$$
-\beta_1=\frac{4+\lambda}{2}.
-$$
+def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
 
-
      Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you.
      
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+
+X = np.array( [ [ 2, 0], [0, 1], [0,0]])
+y = np.array( [4, 2, 3])
+
+
+# matrix inversion to find beta
+OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y
+print(OLSbeta)
+# and then make the prediction
+ytildeOLS = X @ OLSbeta
+print("Training MSE for OLS")
+print(MSE(y,ytildeOLS))
+ypredictOLS = X @ OLSbeta
+
+# Repeat now for Ridge regression and various values of the regularization parameter
+I = np.eye(2,2)
+# Decide which values of lambda to use
+nlambdas = 100
+MSEPredict = np.zeros(nlambdas)
+lambdas = np.logspace(-4, 4, nlambdas)
+for i in range(nlambdas):
+    lmb = lambdas[i]
+    Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
+#    print(Ridgebeta)
+    # and then make the prediction
+    ypredictRidge = X @ Ridgebeta
+    MSEPredict[i] = MSE(y,ypredictRidge)
+#    print(MSEPredict[i])
+    # Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Train')
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      + +

      We see here that we reach a plateau. What is actually happening?

      @@ -347,7 +399,7 @@ $$

    3139. 35
    3140. 36
    3141. ...
      3142. -
    3142. 60
      3143. +
    3143. 59
    3144. »
      3145. diff --git a/doc/pub/week36/html/._week36-bs027.html b/doc/pub/week36/html/._week36-bs027.html index f4c61bc49..e01086037 100644 --- a/doc/pub/week36/html/._week36-bs027.html +++ b/doc/pub/week36/html/._week36-bs027.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    3145. Plans for week 36
      3146. -
    3146. Thursday September 8
      3147. -
    3147. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3148. -
    3148. Linear Regression and the SVD
      3149. -
    3149. What does it mean?
      3150. -
    3150. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3151. -
    3151. Code for SVD and Inversion of Matrices
      3152. -
    3152. Inverse of Rectangular Matrix
      3153. -
    3153. Ridge and LASSO Regression
      3154. -
    3154. From OLS to Ridge and Lasso
      3155. -
    3155. Deriving the Ridge Regression Equations
      3156. -
    3156. Note on Scikit-Learn
      3157. -
    3157. Comparison with OLS
      3158. -
    3158. SVD analysis
      3159. -
    3159. Interpreting the Ridge results
      3160. -
    3160. More interpretations
      3161. -
    3161. Deriving the Lasso Regression Equations
      3162. -
    3162. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3163. -
    3163. Ridge Regression
      3164. -
    3164. Lasso Regression
      3165. -
    3165. Yet another Example
      3166. -
    3166. The OLS case
      3167. -
    3167. The Ridge case
      3168. -
    3168. Writing the Cost Function
      3169. -
    3169. Lasso case
      3170. -
    3170. The first Case
      3171. -
    3171. Simple code for solving the above problem
      3172. -
    3172. With Lasso Regression
      3173. -
    3173. Another Example, now with a polynomial fit
      3174. -
    3174. To think about, first part
      3175. -
    3175. More thinking
      3176. -
    3176. Still thinking
      3177. -
    3177. What does centering (subtracting the mean values) mean mathematically?
      3178. -
    3178. Further Manipulations
      3179. -
    3179. Wrapping it up
      3180. -
    3180. Linear Regression code, Intercept handling first
      3181. -
    3181. Code Examples
      3182. -
    3182. Taking out the mean
      3183. -
    3183. Friday September 9
      3184. -
    3184. Linking the regression analysis with a statistical interpretation
      3185. -
    3185. Assumptions made
      3186. -
    3186. Expectation value and variance
      3187. -
    3187. Expectation value and variance for \( \boldsymbol{\beta} \)
      3188. -
    3188. Deriving OLS from a probability distribution
      3189. -
    3189. Independent and Identically Distrubuted (iid)
      3190. -
    3190. Maximum Likelihood Estimation (MLE)
      3191. -
    3191. A new Cost Function
      3192. -
    3192. More basic Statistics and Bayes' theorem
      3193. -
    3193. Marginal Probability
      3194. -
    3194. Conditional Probability
      3195. -
    3195. Bayes' Theorem
      3196. -
    3196. Interpretations of Bayes' Theorem
      3197. -
    3197. Example of Usage of Bayes' theorem
      3198. -
    3198. Doing it correctly
      3199. -
    3199. Bayes' Theorem and Ridge and Lasso Regression
      3200. -
    3200. Test Function for what happens with OLS, Ridge and Lasso
      3201. -
    3201. Invoking Bayes' theorem
      3202. -
    3202. Ridge and Bayes
      3203. -
    3203. Lasso and Bayes
      3204. -
    3204. Exercise 1: mean values and variances in linear regression
      3205. -
    3205. Exercise 2: Adding Ridge and Lasso Regression
      3206. +
    3206. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3207. +
    3207. Linear Regression and the SVD
      3208. +
    3208. What does it mean?
      3209. +
    3209. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3210. +
    3210. Code for SVD and Inversion of Matrices
      3211. +
    3211. Inverse of Rectangular Matrix
      3212. +
    3212. Ridge and LASSO Regression
      3213. +
    3213. From OLS to Ridge and Lasso
      3214. +
    3214. Deriving the Ridge Regression Equations
      3215. +
    3215. Note on Scikit-Learn
      3216. +
    3216. Comparison with OLS
      3217. +
    3217. SVD analysis
      3218. +
    3218. Interpreting the Ridge results
      3219. +
    3219. More interpretations
      3220. +
    3220. Deriving the Lasso Regression Equations
      3221. +
    3221. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3222. +
    3222. Ridge Regression
      3223. +
    3223. Lasso Regression
      3224. +
    3224. Yet another Example
      3225. +
    3225. The OLS case
      3226. +
    3226. The Ridge case
      3227. +
    3227. Writing the Cost Function
      3228. +
    3228. Lasso case
      3229. +
    3229. The first Case
      3230. +
    3230. Simple code for solving the above problem
      3231. +
    3231. With Lasso Regression
      3232. +
    3232. Another Example, now with a polynomial fit
      3233. +
    3233. To think about, first part
      3234. +
    3234. More thinking
      3235. +
    3235. Still thinking
      3236. +
    3236. What does centering (subtracting the mean values) mean mathematically?
      3237. +
    3237. Further Manipulations
      3238. +
    3238. Wrapping it up
      3239. +
    3239. Linear Regression code, Intercept handling first
      3240. +
    3240. Code Examples
      3241. +
    3241. Taking out the mean
      3242. +
    3242. Friday September 9
      3243. +
    3243. Linking the regression analysis with a statistical interpretation
      3244. +
    3244. Assumptions made
      3245. +
    3245. Expectation value and variance
      3246. +
    3246. Expectation value and variance for \( \boldsymbol{\beta} \)
      3247. +
    3247. Deriving OLS from a probability distribution
      3248. +
    3248. Independent and Identically Distrubuted (iid)
      3249. +
    3249. Maximum Likelihood Estimation (MLE)
      3250. +
    3250. A new Cost Function
      3251. +
    3251. More basic Statistics and Bayes' theorem
      3252. +
    3252. Marginal Probability
      3253. +
    3253. Conditional Probability
      3254. +
    3254. Bayes' Theorem
      3255. +
    3255. Interpretations of Bayes' Theorem
      3256. +
    3256. Example of Usage of Bayes' theorem
      3257. +
    3257. Doing it correctly
      3258. +
    3258. Bayes' Theorem and Ridge and Lasso Regression
      3259. +
    3259. Test Function for what happens with OLS, Ridge and Lasso
      3260. +
    3260. Invoking Bayes' theorem
      3261. +
    3261. Ridge and Bayes
      3262. +
    3262. Lasso and Bayes
      3263. +
    3263. Exercise 1: mean values and variances in linear regression
      3264. +
    3264. Exercise 2: Adding Ridge and Lasso Regression
      3265. @@ -297,11 +295,7 @@ MathJax.Hub.Config({

       

       

       

      -

      Simple code for solving the above problem

      - -

      Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values.

      - -

      First we study and compare the OLS and Ridge results. The next code compares all three methods.

      +

      With Lasso Regression

      @@ -314,6 +308,7 @@ MathJax.Hub.Config({ import numpy as np import pandas as pd import matplotlib.pyplot as plt +from sklearn import linear_model def R2(y_data, y_model): return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2) @@ -342,19 +337,25 @@ ypredictOLS = X = np.eye(2,2) # Decide which values of lambda to use nlambdas = 100 -MSEPredict = np.zeros(nlambdas) +MSERidgePredict = np.zeros(nlambdas) +MSELassoPredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 4, nlambdas) for i in range(nlambdas): lmb = lambdas[i] Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y -# print(Ridgebeta) + print(Ridgebeta) # and then make the prediction ypredictRidge = X @ Ridgebeta - MSEPredict[i] = MSE(y,ypredictRidge) -# print(MSEPredict[i]) - # Now plot the results + MSERidgePredict[i] = MSE(y,ypredictRidge) + RegLasso = linear_model.Lasso(lmb) + RegLasso.fit(X,y) + ypredictLasso = RegLasso.predict(X) + print(RegLasso.coef_) + MSELassoPredict[i] = MSE(y,ypredictLasso) +# Now plot the results plt.figure() -plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Train') +plt.plot(np.log10(lambdas), MSERidgePredict, 'r--', label = 'MSE Ridge Train') +plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Train') plt.xlabel('log10(lambda)') plt.ylabel('MSE') plt.legend() @@ -374,7 +375,6 @@ plt.show() -

      We see here that we reach a plateau. What is actually happening?

      @@ -401,7 +401,7 @@ plt.show()

    3265. 36
    3266. 37
    3267. ...
      3268. -
    3268. 60
      3269. +
    3269. 59
    3270. »
      3271. diff --git a/doc/pub/week36/html/._week36-bs028.html b/doc/pub/week36/html/._week36-bs028.html index 6a12b712a..6dec31562 100644 --- a/doc/pub/week36/html/._week36-bs028.html +++ b/doc/pub/week36/html/._week36-bs028.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    3271. Plans for week 36
      3272. -
    3272. Thursday September 8
      3273. -
    3273. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3274. -
    3274. Linear Regression and the SVD
      3275. -
    3275. What does it mean?
      3276. -
    3276. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3277. -
    3277. Code for SVD and Inversion of Matrices
      3278. -
    3278. Inverse of Rectangular Matrix
      3279. -
    3279. Ridge and LASSO Regression
      3280. -
    3280. From OLS to Ridge and Lasso
      3281. -
    3281. Deriving the Ridge Regression Equations
      3282. -
    3282. Note on Scikit-Learn
      3283. -
    3283. Comparison with OLS
      3284. -
    3284. SVD analysis
      3285. -
    3285. Interpreting the Ridge results
      3286. -
    3286. More interpretations
      3287. -
    3287. Deriving the Lasso Regression Equations
      3288. -
    3288. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3289. -
    3289. Ridge Regression
      3290. -
    3290. Lasso Regression
      3291. -
    3291. Yet another Example
      3292. -
    3292. The OLS case
      3293. -
    3293. The Ridge case
      3294. -
    3294. Writing the Cost Function
      3295. -
    3295. Lasso case
      3296. -
    3296. The first Case
      3297. -
    3297. Simple code for solving the above problem
      3298. -
    3298. With Lasso Regression
      3299. -
    3299. Another Example, now with a polynomial fit
      3300. -
    3300. To think about, first part
      3301. -
    3301. More thinking
      3302. -
    3302. Still thinking
      3303. -
    3303. What does centering (subtracting the mean values) mean mathematically?
      3304. -
    3304. Further Manipulations
      3305. -
    3305. Wrapping it up
      3306. -
    3306. Linear Regression code, Intercept handling first
      3307. -
    3307. Code Examples
      3308. -
    3308. Taking out the mean
      3309. -
    3309. Friday September 9
      3310. -
    3310. Linking the regression analysis with a statistical interpretation
      3311. -
    3311. Assumptions made
      3312. -
    3312. Expectation value and variance
      3313. -
    3313. Expectation value and variance for \( \boldsymbol{\beta} \)
      3314. -
    3314. Deriving OLS from a probability distribution
      3315. -
    3315. Independent and Identically Distrubuted (iid)
      3316. -
    3316. Maximum Likelihood Estimation (MLE)
      3317. -
    3317. A new Cost Function
      3318. -
    3318. More basic Statistics and Bayes' theorem
      3319. -
    3319. Marginal Probability
      3320. -
    3320. Conditional Probability
      3321. -
    3321. Bayes' Theorem
      3322. -
    3322. Interpretations of Bayes' Theorem
      3323. -
    3323. Example of Usage of Bayes' theorem
      3324. -
    3324. Doing it correctly
      3325. -
    3325. Bayes' Theorem and Ridge and Lasso Regression
      3326. -
    3326. Test Function for what happens with OLS, Ridge and Lasso
      3327. -
    3327. Invoking Bayes' theorem
      3328. -
    3328. Ridge and Bayes
      3329. -
    3329. Lasso and Bayes
      3330. -
    3330. Exercise 1: mean values and variances in linear regression
      3331. -
    3331. Exercise 2: Adding Ridge and Lasso Regression
      3332. +
    3332. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3333. +
    3333. Linear Regression and the SVD
      3334. +
    3334. What does it mean?
      3335. +
    3335. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3336. +
    3336. Code for SVD and Inversion of Matrices
      3337. +
    3337. Inverse of Rectangular Matrix
      3338. +
    3338. Ridge and LASSO Regression
      3339. +
    3339. From OLS to Ridge and Lasso
      3340. +
    3340. Deriving the Ridge Regression Equations
      3341. +
    3341. Note on Scikit-Learn
      3342. +
    3342. Comparison with OLS
      3343. +
    3343. SVD analysis
      3344. +
    3344. Interpreting the Ridge results
      3345. +
    3345. More interpretations
      3346. +
    3346. Deriving the Lasso Regression Equations
      3347. +
    3347. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3348. +
    3348. Ridge Regression
      3349. +
    3349. Lasso Regression
      3350. +
    3350. Yet another Example
      3351. +
    3351. The OLS case
      3352. +
    3352. The Ridge case
      3353. +
    3353. Writing the Cost Function
      3354. +
    3354. Lasso case
      3355. +
    3355. The first Case
      3356. +
    3356. Simple code for solving the above problem
      3357. +
    3357. With Lasso Regression
      3358. +
    3358. Another Example, now with a polynomial fit
      3359. +
    3359. To think about, first part
      3360. +
    3360. More thinking
      3361. +
    3361. Still thinking
      3362. +
    3362. What does centering (subtracting the mean values) mean mathematically?
      3363. +
    3363. Further Manipulations
      3364. +
    3364. Wrapping it up
      3365. +
    3365. Linear Regression code, Intercept handling first
      3366. +
    3366. Code Examples
      3367. +
    3367. Taking out the mean
      3368. +
    3368. Friday September 9
      3369. +
    3369. Linking the regression analysis with a statistical interpretation
      3370. +
    3370. Assumptions made
      3371. +
    3371. Expectation value and variance
      3372. +
    3372. Expectation value and variance for \( \boldsymbol{\beta} \)
      3373. +
    3373. Deriving OLS from a probability distribution
      3374. +
    3374. Independent and Identically Distrubuted (iid)
      3375. +
    3375. Maximum Likelihood Estimation (MLE)
      3376. +
    3376. A new Cost Function
      3377. +
    3377. More basic Statistics and Bayes' theorem
      3378. +
    3378. Marginal Probability
      3379. +
    3379. Conditional Probability
      3380. +
    3380. Bayes' Theorem
      3381. +
    3381. Interpretations of Bayes' Theorem
      3382. +
    3382. Example of Usage of Bayes' theorem
      3383. +
    3383. Doing it correctly
      3384. +
    3384. Bayes' Theorem and Ridge and Lasso Regression
      3385. +
    3385. Test Function for what happens with OLS, Ridge and Lasso
      3386. +
    3386. Invoking Bayes' theorem
      3387. +
    3387. Ridge and Bayes
      3388. +
    3388. Lasso and Bayes
      3389. +
    3389. Exercise 1: mean values and variances in linear regression
      3390. +
    3390. Exercise 2: Adding Ridge and Lasso Regression
      3391. @@ -297,7 +295,7 @@ MathJax.Hub.Config({

       

       

       

      -

      With Lasso Regression

      +

      Another Example, now with a polynomial fit

      @@ -310,6 +308,7 @@ MathJax.Hub.Config({ import numpy as np import pandas as pd import matplotlib.pyplot as plt +from sklearn.model_selection import train_test_split from sklearn import linear_model def R2(y_data, y_model): @@ -321,43 +320,64 @@ MathJax.Hub.Config({ # A seed just to ensure that the random numbers are the same for every run. # Useful for eventual debugging. +np.random.seed(3155) -X = np.array( [ [ 2, 0], [0, 1], [0,0]]) -y = np.array( [4, 2, 3]) +x = np.random.rand(100) +y = 2.0+5*x*x+0.1*np.random.randn(100) +# number of features p (here degree of polynomial +p = 3 +# The design matrix now as function of a given polynomial +X = np.zeros((len(x),p)) +X[:,0] = 1.0 +X[:,1] = x +X[:,2] = x*x +# We split the data in test and training data +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) # matrix inversion to find beta -OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y +OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train print(OLSbeta) # and then make the prediction -ytildeOLS = X @ OLSbeta +ytildeOLS = X_train @ OLSbeta print("Training MSE for OLS") -print(MSE(y,ytildeOLS)) -ypredictOLS = X @ OLSbeta +print(MSE(y_train,ytildeOLS)) +ypredictOLS = X_test @ OLSbeta +print("Test MSE OLS") +print(MSE(y_test,ypredictOLS)) -# Repeat now for Ridge regression and various values of the regularization parameter -I = np.eye(2,2) +# Repeat now for Lasso and Ridge regression and various values of the regularization parameter +I = np.eye(p,p) # Decide which values of lambda to use nlambdas = 100 -MSERidgePredict = np.zeros(nlambdas) +MSEPredict = np.zeros(nlambdas) +MSETrain = np.zeros(nlambdas) MSELassoPredict = np.zeros(nlambdas) +MSELassoTrain = np.zeros(nlambdas) lambdas = np.logspace(-4, 4, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y - print(Ridgebeta) - # and then make the prediction - ypredictRidge = X @ Ridgebeta - MSERidgePredict[i] = MSE(y,ypredictRidge) + Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + # include lasso using Scikit-Learn RegLasso = linear_model.Lasso(lmb) - RegLasso.fit(X,y) - ypredictLasso = RegLasso.predict(X) - print(RegLasso.coef_) - MSELassoPredict[i] = MSE(y,ypredictLasso) + RegLasso.fit(X_train,y_train) + # and then make the prediction + ytildeRidge = X_train @ Ridgebeta + ypredictRidge = X_test @ Ridgebeta + ytildeLasso = RegLasso.predict(X_train) + ypredictLasso = RegLasso.predict(X_test) + MSEPredict[i] = MSE(y_test,ypredictRidge) + MSETrain[i] = MSE(y_train,ytildeRidge) + MSELassoPredict[i] = MSE(y_test,ypredictLasso) + MSELassoTrain[i] = MSE(y_train,ytildeLasso) + # Now plot the results plt.figure() -plt.plot(np.log10(lambdas), MSERidgePredict, 'r--', label = 'MSE Ridge Train') -plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Train') +plt.plot(np.log10(lambdas), MSETrain, label = 'MSE Ridge train') +plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Test') +plt.plot(np.log10(lambdas), MSELassoTrain, label = 'MSE Lasso train') +plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Test') + plt.xlabel('log10(lambda)') plt.ylabel('MSE') plt.legend() @@ -403,7 +423,7 @@ plt.show()
    3391. 37
    3392. 38
    3393. ...
      3394. -
    3394. 60
      3395. +
    3395. 59
    3396. »
      3397. diff --git a/doc/pub/week36/html/._week36-bs029.html b/doc/pub/week36/html/._week36-bs029.html index 1d0525f83..c649c8408 100644 --- a/doc/pub/week36/html/._week36-bs029.html +++ b/doc/pub/week36/html/._week36-bs029.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    3397. Plans for week 36
      3398. -
    3398. Thursday September 8
      3399. -
    3399. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3400. -
    3400. Linear Regression and the SVD
      3401. -
    3401. What does it mean?
      3402. -
    3402. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3403. -
    3403. Code for SVD and Inversion of Matrices
      3404. -
    3404. Inverse of Rectangular Matrix
      3405. -
    3405. Ridge and LASSO Regression
      3406. -
    3406. From OLS to Ridge and Lasso
      3407. -
    3407. Deriving the Ridge Regression Equations
      3408. -
    3408. Note on Scikit-Learn
      3409. -
    3409. Comparison with OLS
      3410. -
    3410. SVD analysis
      3411. -
    3411. Interpreting the Ridge results
      3412. -
    3412. More interpretations
      3413. -
    3413. Deriving the Lasso Regression Equations
      3414. -
    3414. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3415. -
    3415. Ridge Regression
      3416. -
    3416. Lasso Regression
      3417. -
    3417. Yet another Example
      3418. -
    3418. The OLS case
      3419. -
    3419. The Ridge case
      3420. -
    3420. Writing the Cost Function
      3421. -
    3421. Lasso case
      3422. -
    3422. The first Case
      3423. -
    3423. Simple code for solving the above problem
      3424. -
    3424. With Lasso Regression
      3425. -
    3425. Another Example, now with a polynomial fit
      3426. -
    3426. To think about, first part
      3427. -
    3427. More thinking
      3428. -
    3428. Still thinking
      3429. -
    3429. What does centering (subtracting the mean values) mean mathematically?
      3430. -
    3430. Further Manipulations
      3431. -
    3431. Wrapping it up
      3432. -
    3432. Linear Regression code, Intercept handling first
      3433. -
    3433. Code Examples
      3434. -
    3434. Taking out the mean
      3435. -
    3435. Friday September 9
      3436. -
    3436. Linking the regression analysis with a statistical interpretation
      3437. -
    3437. Assumptions made
      3438. -
    3438. Expectation value and variance
      3439. -
    3439. Expectation value and variance for \( \boldsymbol{\beta} \)
      3440. -
    3440. Deriving OLS from a probability distribution
      3441. -
    3441. Independent and Identically Distrubuted (iid)
      3442. -
    3442. Maximum Likelihood Estimation (MLE)
      3443. -
    3443. A new Cost Function
      3444. -
    3444. More basic Statistics and Bayes' theorem
      3445. -
    3445. Marginal Probability
      3446. -
    3446. Conditional Probability
      3447. -
    3447. Bayes' Theorem
      3448. -
    3448. Interpretations of Bayes' Theorem
      3449. -
    3449. Example of Usage of Bayes' theorem
      3450. -
    3450. Doing it correctly
      3451. -
    3451. Bayes' Theorem and Ridge and Lasso Regression
      3452. -
    3452. Test Function for what happens with OLS, Ridge and Lasso
      3453. -
    3453. Invoking Bayes' theorem
      3454. -
    3454. Ridge and Bayes
      3455. -
    3455. Lasso and Bayes
      3456. -
    3456. Exercise 1: mean values and variances in linear regression
      3457. -
    3457. Exercise 2: Adding Ridge and Lasso Regression
      3458. +
    3458. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3459. +
    3459. Linear Regression and the SVD
      3460. +
    3460. What does it mean?
      3461. +
    3461. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3462. +
    3462. Code for SVD and Inversion of Matrices
      3463. +
    3463. Inverse of Rectangular Matrix
      3464. +
    3464. Ridge and LASSO Regression
      3465. +
    3465. From OLS to Ridge and Lasso
      3466. +
    3466. Deriving the Ridge Regression Equations
      3467. +
    3467. Note on Scikit-Learn
      3468. +
    3468. Comparison with OLS
      3469. +
    3469. SVD analysis
      3470. +
    3470. Interpreting the Ridge results
      3471. +
    3471. More interpretations
      3472. +
    3472. Deriving the Lasso Regression Equations
      3473. +
    3473. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3474. +
    3474. Ridge Regression
      3475. +
    3475. Lasso Regression
      3476. +
    3476. Yet another Example
      3477. +
    3477. The OLS case
      3478. +
    3478. The Ridge case
      3479. +
    3479. Writing the Cost Function
      3480. +
    3480. Lasso case
      3481. +
    3481. The first Case
      3482. +
    3482. Simple code for solving the above problem
      3483. +
    3483. With Lasso Regression
      3484. +
    3484. Another Example, now with a polynomial fit
      3485. +
    3485. To think about, first part
      3486. +
    3486. More thinking
      3487. +
    3487. Still thinking
      3488. +
    3488. What does centering (subtracting the mean values) mean mathematically?
      3489. +
    3489. Further Manipulations
      3490. +
    3490. Wrapping it up
      3491. +
    3491. Linear Regression code, Intercept handling first
      3492. +
    3492. Code Examples
      3493. +
    3493. Taking out the mean
      3494. +
    3494. Friday September 9
      3495. +
    3495. Linking the regression analysis with a statistical interpretation
      3496. +
    3496. Assumptions made
      3497. +
    3497. Expectation value and variance
      3498. +
    3498. Expectation value and variance for \( \boldsymbol{\beta} \)
      3499. +
    3499. Deriving OLS from a probability distribution
      3500. +
    3500. Independent and Identically Distrubuted (iid)
      3501. +
    3501. Maximum Likelihood Estimation (MLE)
      3502. +
    3502. A new Cost Function
      3503. +
    3503. More basic Statistics and Bayes' theorem
      3504. +
    3504. Marginal Probability
      3505. +
    3505. Conditional Probability
      3506. +
    3506. Bayes' Theorem
      3507. +
    3507. Interpretations of Bayes' Theorem
      3508. +
    3508. Example of Usage of Bayes' theorem
      3509. +
    3509. Doing it correctly
      3510. +
    3510. Bayes' Theorem and Ridge and Lasso Regression
      3511. +
    3511. Test Function for what happens with OLS, Ridge and Lasso
      3512. +
    3512. Invoking Bayes' theorem
      3513. +
    3513. Ridge and Bayes
      3514. +
    3514. Lasso and Bayes
      3515. +
    3515. Exercise 1: mean values and variances in linear regression
      3516. +
    3516. Exercise 2: Adding Ridge and Lasso Regression
      3517. @@ -297,108 +295,28 @@ MathJax.Hub.Config({

       

       

       

      -

      Another Example, now with a polynomial fit

      +

      To think about, first part

      +

      When you are comparing your own code with for example Scikit-Learn's +library, there are some technicalities to keep in mind. The examples +here demonstrate some of these aspects with potential pitfalls. +

      - -
      -
      -
      -
      -
      - 
      import os
-import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-from sklearn import linear_model
-
-def R2(y_data, y_model):
-    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-def MSE(y_data,y_model):
-    n = np.size(y_model)
-    return np.sum((y_data-y_model)**2)/n
-
-
-# A seed just to ensure that the random numbers are the same for every run.
-# Useful for eventual debugging.
-np.random.seed(3155)
-
-x = np.random.rand(100)
-y = 2.0+5*x*x+0.1*np.random.randn(100)
-
-# number of features p (here degree of polynomial
-p = 3
-#  The design matrix now as function of a given polynomial
-X = np.zeros((len(x),p))
-X[:,0] = 1.0
-X[:,1] = x
-X[:,2] = x*x
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-
-# matrix inversion to find beta
-OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-print(OLSbeta)
-# and then make the prediction
-ytildeOLS = X_train @ OLSbeta
-print("Training MSE for OLS")
-print(MSE(y_train,ytildeOLS))
-ypredictOLS = X_test @ OLSbeta
-print("Test MSE OLS")
-print(MSE(y_test,ypredictOLS))
-
-# Repeat now for Lasso and Ridge regression and various values of the regularization parameter
-I = np.eye(p,p)
-# Decide which values of lambda to use
-nlambdas = 100
-MSEPredict = np.zeros(nlambdas)
-MSETrain = np.zeros(nlambdas)
-MSELassoPredict = np.zeros(nlambdas)
-MSELassoTrain = np.zeros(nlambdas)
-lambdas = np.logspace(-4, 4, nlambdas)
-for i in range(nlambdas):
-    lmb = lambdas[i]
-    Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
-    # include lasso using Scikit-Learn
-    RegLasso = linear_model.Lasso(lmb)
-    RegLasso.fit(X_train,y_train)
-    # and then make the prediction
-    ytildeRidge = X_train @ Ridgebeta
-    ypredictRidge = X_test @ Ridgebeta
-    ytildeLasso = RegLasso.predict(X_train)
-    ypredictLasso = RegLasso.predict(X_test)
-    MSEPredict[i] = MSE(y_test,ypredictRidge)
-    MSETrain[i] = MSE(y_train,ytildeRidge)
-    MSELassoPredict[i] = MSE(y_test,ypredictLasso)
-    MSELassoTrain[i] = MSE(y_train,ytildeLasso)
-
-# Now plot the results
-plt.figure()
-plt.plot(np.log10(lambdas), MSETrain, label = 'MSE Ridge train')
-plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Test')
-plt.plot(np.log10(lambdas), MSELassoTrain, label = 'MSE Lasso train')
-plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Test')
-
-plt.xlabel('log10(lambda)')
-plt.ylabel('MSE')
-plt.legend()
-plt.show()
-
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      +

      The discussion here focuses on the role of the intercept, how we can +set up the design matrix, what scaling we should use and other topics +which tend confuse us. +

      +

      The intercept can be interpreted as the expected value of our +target/output variables when all other predictors are set to zero. +Thus, if we cannot assume that the expected outputs/targets are zero +when all predictors are zero (the columns in the design matrix), it +may be a bad idea to implement a model which penalizes the intercept. +Furthermore, in for example Ridge and Lasso regression, the default solutions +from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters +\( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and +\( \boldsymbol{X} \) are zero centered, that is we subtract the mean values. +

      @@ -425,7 +343,7 @@ plt.show()

    3517. 38
    3518. 39
    3519. ...
      3520. -
    3520. 60
      3521. +
    3521. 59
    3522. »
      3523. diff --git a/doc/pub/week36/html/._week36-bs030.html b/doc/pub/week36/html/._week36-bs030.html index ec0271389..0e6419f88 100644 --- a/doc/pub/week36/html/._week36-bs030.html +++ b/doc/pub/week36/html/._week36-bs030.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    3523. Plans for week 36
      3524. -
    3524. Thursday September 8
      3525. -
    3525. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3526. -
    3526. Linear Regression and the SVD
      3527. -
    3527. What does it mean?
      3528. -
    3528. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3529. -
    3529. Code for SVD and Inversion of Matrices
      3530. -
    3530. Inverse of Rectangular Matrix
      3531. -
    3531. Ridge and LASSO Regression
      3532. -
    3532. From OLS to Ridge and Lasso
      3533. -
    3533. Deriving the Ridge Regression Equations
      3534. -
    3534. Note on Scikit-Learn
      3535. -
    3535. Comparison with OLS
      3536. -
    3536. SVD analysis
      3537. -
    3537. Interpreting the Ridge results
      3538. -
    3538. More interpretations
      3539. -
    3539. Deriving the Lasso Regression Equations
      3540. -
    3540. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3541. -
    3541. Ridge Regression
      3542. -
    3542. Lasso Regression
      3543. -
    3543. Yet another Example
      3544. -
    3544. The OLS case
      3545. -
    3545. The Ridge case
      3546. -
    3546. Writing the Cost Function
      3547. -
    3547. Lasso case
      3548. -
    3548. The first Case
      3549. -
    3549. Simple code for solving the above problem
      3550. -
    3550. With Lasso Regression
      3551. -
    3551. Another Example, now with a polynomial fit
      3552. -
    3552. To think about, first part
      3553. -
    3553. More thinking
      3554. -
    3554. Still thinking
      3555. -
    3555. What does centering (subtracting the mean values) mean mathematically?
      3556. -
    3556. Further Manipulations
      3557. -
    3557. Wrapping it up
      3558. -
    3558. Linear Regression code, Intercept handling first
      3559. -
    3559. Code Examples
      3560. -
    3560. Taking out the mean
      3561. -
    3561. Friday September 9
      3562. -
    3562. Linking the regression analysis with a statistical interpretation
      3563. -
    3563. Assumptions made
      3564. -
    3564. Expectation value and variance
      3565. -
    3565. Expectation value and variance for \( \boldsymbol{\beta} \)
      3566. -
    3566. Deriving OLS from a probability distribution
      3567. -
    3567. Independent and Identically Distrubuted (iid)
      3568. -
    3568. Maximum Likelihood Estimation (MLE)
      3569. -
    3569. A new Cost Function
      3570. -
    3570. More basic Statistics and Bayes' theorem
      3571. -
    3571. Marginal Probability
      3572. -
    3572. Conditional Probability
      3573. -
    3573. Bayes' Theorem
      3574. -
    3574. Interpretations of Bayes' Theorem
      3575. -
    3575. Example of Usage of Bayes' theorem
      3576. -
    3576. Doing it correctly
      3577. -
    3577. Bayes' Theorem and Ridge and Lasso Regression
      3578. -
    3578. Test Function for what happens with OLS, Ridge and Lasso
      3579. -
    3579. Invoking Bayes' theorem
      3580. -
    3580. Ridge and Bayes
      3581. -
    3581. Lasso and Bayes
      3582. -
    3582. Exercise 1: mean values and variances in linear regression
      3583. -
    3583. Exercise 2: Adding Ridge and Lasso Regression
      3584. +
    3584. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3585. +
    3585. Linear Regression and the SVD
      3586. +
    3586. What does it mean?
      3587. +
    3587. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3588. +
    3588. Code for SVD and Inversion of Matrices
      3589. +
    3589. Inverse of Rectangular Matrix
      3590. +
    3590. Ridge and LASSO Regression
      3591. +
    3591. From OLS to Ridge and Lasso
      3592. +
    3592. Deriving the Ridge Regression Equations
      3593. +
    3593. Note on Scikit-Learn
      3594. +
    3594. Comparison with OLS
      3595. +
    3595. SVD analysis
      3596. +
    3596. Interpreting the Ridge results
      3597. +
    3597. More interpretations
      3598. +
    3598. Deriving the Lasso Regression Equations
      3599. +
    3599. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3600. +
    3600. Ridge Regression
      3601. +
    3601. Lasso Regression
      3602. +
    3602. Yet another Example
      3603. +
    3603. The OLS case
      3604. +
    3604. The Ridge case
      3605. +
    3605. Writing the Cost Function
      3606. +
    3606. Lasso case
      3607. +
    3607. The first Case
      3608. +
    3608. Simple code for solving the above problem
      3609. +
    3609. With Lasso Regression
      3610. +
    3610. Another Example, now with a polynomial fit
      3611. +
    3611. To think about, first part
      3612. +
    3612. More thinking
      3613. +
    3613. Still thinking
      3614. +
    3614. What does centering (subtracting the mean values) mean mathematically?
      3615. +
    3615. Further Manipulations
      3616. +
    3616. Wrapping it up
      3617. +
    3617. Linear Regression code, Intercept handling first
      3618. +
    3618. Code Examples
      3619. +
    3619. Taking out the mean
      3620. +
    3620. Friday September 9
      3621. +
    3621. Linking the regression analysis with a statistical interpretation
      3622. +
    3622. Assumptions made
      3623. +
    3623. Expectation value and variance
      3624. +
    3624. Expectation value and variance for \( \boldsymbol{\beta} \)
      3625. +
    3625. Deriving OLS from a probability distribution
      3626. +
    3626. Independent and Identically Distrubuted (iid)
      3627. +
    3627. Maximum Likelihood Estimation (MLE)
      3628. +
    3628. A new Cost Function
      3629. +
    3629. More basic Statistics and Bayes' theorem
      3630. +
    3630. Marginal Probability
      3631. +
    3631. Conditional Probability
      3632. +
    3632. Bayes' Theorem
      3633. +
    3633. Interpretations of Bayes' Theorem
      3634. +
    3634. Example of Usage of Bayes' theorem
      3635. +
    3635. Doing it correctly
      3636. +
    3636. Bayes' Theorem and Ridge and Lasso Regression
      3637. +
    3637. Test Function for what happens with OLS, Ridge and Lasso
      3638. +
    3638. Invoking Bayes' theorem
      3639. +
    3639. Ridge and Bayes
      3640. +
    3640. Lasso and Bayes
      3641. +
    3641. Exercise 1: mean values and variances in linear regression
      3642. +
    3642. Exercise 2: Adding Ridge and Lasso Regression
      3643. @@ -297,27 +295,35 @@ MathJax.Hub.Config({

       

       

       

      -

      To think about, first part

      +

      More thinking

      -

      When you are comparing your own code with for example Scikit-Learn's -library, there are some technicalities to keep in mind. The examples -here demonstrate some of these aspects with potential pitfalls. +

      If our predictors represent different scales, then it is important to +standardize the design matrix \( \boldsymbol{X} \) by subtracting the mean of each +column from the corresponding column and dividing the column with its +standard deviation. Most machine learning libraries do this as a default. This means that if you compare your code with the results from a given library, +the results may differ.

      -

      The discussion here focuses on the role of the intercept, how we can -set up the design matrix, what scaling we should use and other topics -which tend confuse us. +

      The +Standadscaler +function in Scikit-Learn does this for us. For the data sets we +have been studying in our various examples, the data are in many cases +already scaled and there is no need to scale them. You as a user of different machine learning algorithms, should always perform a +survey of your data, with a critical assessment of them in case you need to scale the data.

      -

      The intercept can be interpreted as the expected value of our -target/output variables when all other predictors are set to zero. -Thus, if we cannot assume that the expected outputs/targets are zero -when all predictors are zero (the columns in the design matrix), it -may be a bad idea to implement a model which penalizes the intercept. -Furthermore, in for example Ridge and Lasso regression, the default solutions -from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters -\( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and -\( \boldsymbol{X} \) are zero centered, that is we subtract the mean values. +

      If you need to scale the data, not doing so will give an unfair +penalization of the parameters since their magnitude depends on the +scale of their corresponding predictor. +

      + +

      Suppose as an example that you +you have an input variable given by the heights of different persons. +Human height might be measured in inches or meters or +kilometers. If measured in kilometers, a standard linear regression +model with this predictor would probably give a much bigger +coefficient term, than if measured in millimeters. +This can clearly lead to problems in evaluating the cost/loss functions.

      @@ -345,7 +351,7 @@ from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the

    3643. 39
    3644. 40
    3645. ...
      3646. -
    3646. 60
      3647. +
    3647. 59
    3648. »
      3649. diff --git a/doc/pub/week36/html/._week36-bs031.html b/doc/pub/week36/html/._week36-bs031.html index 045b0cb25..f4b879d80 100644 --- a/doc/pub/week36/html/._week36-bs031.html +++ b/doc/pub/week36/html/._week36-bs031.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    3649. Plans for week 36
      3650. -
    3650. Thursday September 8
      3651. -
    3651. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3652. -
    3652. Linear Regression and the SVD
      3653. -
    3653. What does it mean?
      3654. -
    3654. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3655. -
    3655. Code for SVD and Inversion of Matrices
      3656. -
    3656. Inverse of Rectangular Matrix
      3657. -
    3657. Ridge and LASSO Regression
      3658. -
    3658. From OLS to Ridge and Lasso
      3659. -
    3659. Deriving the Ridge Regression Equations
      3660. -
    3660. Note on Scikit-Learn
      3661. -
    3661. Comparison with OLS
      3662. -
    3662. SVD analysis
      3663. -
    3663. Interpreting the Ridge results
      3664. -
    3664. More interpretations
      3665. -
    3665. Deriving the Lasso Regression Equations
      3666. -
    3666. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3667. -
    3667. Ridge Regression
      3668. -
    3668. Lasso Regression
      3669. -
    3669. Yet another Example
      3670. -
    3670. The OLS case
      3671. -
    3671. The Ridge case
      3672. -
    3672. Writing the Cost Function
      3673. -
    3673. Lasso case
      3674. -
    3674. The first Case
      3675. -
    3675. Simple code for solving the above problem
      3676. -
    3676. With Lasso Regression
      3677. -
    3677. Another Example, now with a polynomial fit
      3678. -
    3678. To think about, first part
      3679. -
    3679. More thinking
      3680. -
    3680. Still thinking
      3681. -
    3681. What does centering (subtracting the mean values) mean mathematically?
      3682. -
    3682. Further Manipulations
      3683. -
    3683. Wrapping it up
      3684. -
    3684. Linear Regression code, Intercept handling first
      3685. -
    3685. Code Examples
      3686. -
    3686. Taking out the mean
      3687. -
    3687. Friday September 9
      3688. -
    3688. Linking the regression analysis with a statistical interpretation
      3689. -
    3689. Assumptions made
      3690. -
    3690. Expectation value and variance
      3691. -
    3691. Expectation value and variance for \( \boldsymbol{\beta} \)
      3692. -
    3692. Deriving OLS from a probability distribution
      3693. -
    3693. Independent and Identically Distrubuted (iid)
      3694. -
    3694. Maximum Likelihood Estimation (MLE)
      3695. -
    3695. A new Cost Function
      3696. -
    3696. More basic Statistics and Bayes' theorem
      3697. -
    3697. Marginal Probability
      3698. -
    3698. Conditional Probability
      3699. -
    3699. Bayes' Theorem
      3700. -
    3700. Interpretations of Bayes' Theorem
      3701. -
    3701. Example of Usage of Bayes' theorem
      3702. -
    3702. Doing it correctly
      3703. -
    3703. Bayes' Theorem and Ridge and Lasso Regression
      3704. -
    3704. Test Function for what happens with OLS, Ridge and Lasso
      3705. -
    3705. Invoking Bayes' theorem
      3706. -
    3706. Ridge and Bayes
      3707. -
    3707. Lasso and Bayes
      3708. -
    3708. Exercise 1: mean values and variances in linear regression
      3709. -
    3709. Exercise 2: Adding Ridge and Lasso Regression
      3710. +
    3710. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3711. +
    3711. Linear Regression and the SVD
      3712. +
    3712. What does it mean?
      3713. +
    3713. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3714. +
    3714. Code for SVD and Inversion of Matrices
      3715. +
    3715. Inverse of Rectangular Matrix
      3716. +
    3716. Ridge and LASSO Regression
      3717. +
    3717. From OLS to Ridge and Lasso
      3718. +
    3718. Deriving the Ridge Regression Equations
      3719. +
    3719. Note on Scikit-Learn
      3720. +
    3720. Comparison with OLS
      3721. +
    3721. SVD analysis
      3722. +
    3722. Interpreting the Ridge results
      3723. +
    3723. More interpretations
      3724. +
    3724. Deriving the Lasso Regression Equations
      3725. +
    3725. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3726. +
    3726. Ridge Regression
      3727. +
    3727. Lasso Regression
      3728. +
    3728. Yet another Example
      3729. +
    3729. The OLS case
      3730. +
    3730. The Ridge case
      3731. +
    3731. Writing the Cost Function
      3732. +
    3732. Lasso case
      3733. +
    3733. The first Case
      3734. +
    3734. Simple code for solving the above problem
      3735. +
    3735. With Lasso Regression
      3736. +
    3736. Another Example, now with a polynomial fit
      3737. +
    3737. To think about, first part
      3738. +
    3738. More thinking
      3739. +
    3739. Still thinking
      3740. +
    3740. What does centering (subtracting the mean values) mean mathematically?
      3741. +
    3741. Further Manipulations
      3742. +
    3742. Wrapping it up
      3743. +
    3743. Linear Regression code, Intercept handling first
      3744. +
    3744. Code Examples
      3745. +
    3745. Taking out the mean
      3746. +
    3746. Friday September 9
      3747. +
    3747. Linking the regression analysis with a statistical interpretation
      3748. +
    3748. Assumptions made
      3749. +
    3749. Expectation value and variance
      3750. +
    3750. Expectation value and variance for \( \boldsymbol{\beta} \)
      3751. +
    3751. Deriving OLS from a probability distribution
      3752. +
    3752. Independent and Identically Distrubuted (iid)
      3753. +
    3753. Maximum Likelihood Estimation (MLE)
      3754. +
    3754. A new Cost Function
      3755. +
    3755. More basic Statistics and Bayes' theorem
      3756. +
    3756. Marginal Probability
      3757. +
    3757. Conditional Probability
      3758. +
    3758. Bayes' Theorem
      3759. +
    3759. Interpretations of Bayes' Theorem
      3760. +
    3760. Example of Usage of Bayes' theorem
      3761. +
    3761. Doing it correctly
      3762. +
    3762. Bayes' Theorem and Ridge and Lasso Regression
      3763. +
    3763. Test Function for what happens with OLS, Ridge and Lasso
      3764. +
    3764. Invoking Bayes' theorem
      3765. +
    3765. Ridge and Bayes
      3766. +
    3766. Lasso and Bayes
      3767. +
    3767. Exercise 1: mean values and variances in linear regression
      3768. +
    3768. Exercise 2: Adding Ridge and Lasso Regression
      3769. @@ -297,36 +295,48 @@ MathJax.Hub.Config({

       

       

       

      -

      More thinking

      +

      Still thinking

      -

      If our predictors represent different scales, then it is important to -standardize the design matrix \( \boldsymbol{X} \) by subtracting the mean of each -column from the corresponding column and dividing the column with its -standard deviation. Most machine learning libraries do this as a default. This means that if you compare your code with the results from a given library, -the results may differ. +

      Keep in mind that when you transform your data set before training a model, the same transformation needs to be done +on your eventual new data set before making a prediction. If we translate this into a Python code, it would could be implemented as follows

      -

      The -Standadscaler -function in Scikit-Learn does this for us. For the data sets we -have been studying in our various examples, the data are in many cases -already scaled and there is no need to scale them. You as a user of different machine learning algorithms, should always perform a -survey of your data, with a critical assessment of them in case you need to scale the data. -

      -

      If you need to scale the data, not doing so will give an unfair -penalization of the parameters since their magnitude depends on the -scale of their corresponding predictor. -

      + +
      +
      +
      +
      +
      + 
      #Model training, we compute the mean value of y and X
+y_train_mean = np.mean(y_train)
+X_train_mean = np.mean(X_train,axis=0)
+X_train = X_train - X_train_mean
+y_train = y_train - y_train_mean
+
+# The we fit our model with the training data
+trained_model = some_model.fit(X_train,y_train)
+
+
+#Model prediction, we need also to transform our data set used for the prediction.
+X_test = X_test - X_train_mean #Use mean from training data
+y_pred = trained_model(X_test)
+y_pred = y_pred + y_train_mean
+
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      -

      Suppose as an example that you -you have an input variable given by the heights of different persons. -Human height might be measured in inches or meters or -kilometers. If measured in kilometers, a standard linear regression -model with this predictor would probably give a much bigger -coefficient term, than if measured in millimeters. -This can clearly lead to problems in evaluating the cost/loss functions. -

      @@ -353,7 +363,7 @@ This can clearly lead to problems in evaluating the cost/loss functions.

    3769. 40
    3770. 41
    3771. ...
      3772. -
    3772. 60
      3773. +
    3773. 59
    3774. »
      3775. diff --git a/doc/pub/week36/html/._week36-bs032.html b/doc/pub/week36/html/._week36-bs032.html index 06111bcce..fde17c13b 100644 --- a/doc/pub/week36/html/._week36-bs032.html +++ b/doc/pub/week36/html/._week36-bs032.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    3775. Plans for week 36
      3776. -
    3776. Thursday September 8
      3777. -
    3777. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3778. -
    3778. Linear Regression and the SVD
      3779. -
    3779. What does it mean?
      3780. -
    3780. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3781. -
    3781. Code for SVD and Inversion of Matrices
      3782. -
    3782. Inverse of Rectangular Matrix
      3783. -
    3783. Ridge and LASSO Regression
      3784. -
    3784. From OLS to Ridge and Lasso
      3785. -
    3785. Deriving the Ridge Regression Equations
      3786. -
    3786. Note on Scikit-Learn
      3787. -
    3787. Comparison with OLS
      3788. -
    3788. SVD analysis
      3789. -
    3789. Interpreting the Ridge results
      3790. -
    3790. More interpretations
      3791. -
    3791. Deriving the Lasso Regression Equations
      3792. -
    3792. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3793. -
    3793. Ridge Regression
      3794. -
    3794. Lasso Regression
      3795. -
    3795. Yet another Example
      3796. -
    3796. The OLS case
      3797. -
    3797. The Ridge case
      3798. -
    3798. Writing the Cost Function
      3799. -
    3799. Lasso case
      3800. -
    3800. The first Case
      3801. -
    3801. Simple code for solving the above problem
      3802. -
    3802. With Lasso Regression
      3803. -
    3803. Another Example, now with a polynomial fit
      3804. -
    3804. To think about, first part
      3805. -
    3805. More thinking
      3806. -
    3806. Still thinking
      3807. -
    3807. What does centering (subtracting the mean values) mean mathematically?
      3808. -
    3808. Further Manipulations
      3809. -
    3809. Wrapping it up
      3810. -
    3810. Linear Regression code, Intercept handling first
      3811. -
    3811. Code Examples
      3812. -
    3812. Taking out the mean
      3813. -
    3813. Friday September 9
      3814. -
    3814. Linking the regression analysis with a statistical interpretation
      3815. -
    3815. Assumptions made
      3816. -
    3816. Expectation value and variance
      3817. -
    3817. Expectation value and variance for \( \boldsymbol{\beta} \)
      3818. -
    3818. Deriving OLS from a probability distribution
      3819. -
    3819. Independent and Identically Distrubuted (iid)
      3820. -
    3820. Maximum Likelihood Estimation (MLE)
      3821. -
    3821. A new Cost Function
      3822. -
    3822. More basic Statistics and Bayes' theorem
      3823. -
    3823. Marginal Probability
      3824. -
    3824. Conditional Probability
      3825. -
    3825. Bayes' Theorem
      3826. -
    3826. Interpretations of Bayes' Theorem
      3827. -
    3827. Example of Usage of Bayes' theorem
      3828. -
    3828. Doing it correctly
      3829. -
    3829. Bayes' Theorem and Ridge and Lasso Regression
      3830. -
    3830. Test Function for what happens with OLS, Ridge and Lasso
      3831. -
    3831. Invoking Bayes' theorem
      3832. -
    3832. Ridge and Bayes
      3833. -
    3833. Lasso and Bayes
      3834. -
    3834. Exercise 1: mean values and variances in linear regression
      3835. -
    3835. Exercise 2: Adding Ridge and Lasso Regression
      3836. +
    3836. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3837. +
    3837. Linear Regression and the SVD
      3838. +
    3838. What does it mean?
      3839. +
    3839. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3840. +
    3840. Code for SVD and Inversion of Matrices
      3841. +
    3841. Inverse of Rectangular Matrix
      3842. +
    3842. Ridge and LASSO Regression
      3843. +
    3843. From OLS to Ridge and Lasso
      3844. +
    3844. Deriving the Ridge Regression Equations
      3845. +
    3845. Note on Scikit-Learn
      3846. +
    3846. Comparison with OLS
      3847. +
    3847. SVD analysis
      3848. +
    3848. Interpreting the Ridge results
      3849. +
    3849. More interpretations
      3850. +
    3850. Deriving the Lasso Regression Equations
      3851. +
    3851. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3852. +
    3852. Ridge Regression
      3853. +
    3853. Lasso Regression
      3854. +
    3854. Yet another Example
      3855. +
    3855. The OLS case
      3856. +
    3856. The Ridge case
      3857. +
    3857. Writing the Cost Function
      3858. +
    3858. Lasso case
      3859. +
    3859. The first Case
      3860. +
    3860. Simple code for solving the above problem
      3861. +
    3861. With Lasso Regression
      3862. +
    3862. Another Example, now with a polynomial fit
      3863. +
    3863. To think about, first part
      3864. +
    3864. More thinking
      3865. +
    3865. Still thinking
      3866. +
    3866. What does centering (subtracting the mean values) mean mathematically?
      3867. +
    3867. Further Manipulations
      3868. +
    3868. Wrapping it up
      3869. +
    3869. Linear Regression code, Intercept handling first
      3870. +
    3870. Code Examples
      3871. +
    3871. Taking out the mean
      3872. +
    3872. Friday September 9
      3873. +
    3873. Linking the regression analysis with a statistical interpretation
      3874. +
    3874. Assumptions made
      3875. +
    3875. Expectation value and variance
      3876. +
    3876. Expectation value and variance for \( \boldsymbol{\beta} \)
      3877. +
    3877. Deriving OLS from a probability distribution
      3878. +
    3878. Independent and Identically Distrubuted (iid)
      3879. +
    3879. Maximum Likelihood Estimation (MLE)
      3880. +
    3880. A new Cost Function
      3881. +
    3881. More basic Statistics and Bayes' theorem
      3882. +
    3882. Marginal Probability
      3883. +
    3883. Conditional Probability
      3884. +
    3884. Bayes' Theorem
      3885. +
    3885. Interpretations of Bayes' Theorem
      3886. +
    3886. Example of Usage of Bayes' theorem
      3887. +
    3887. Doing it correctly
      3888. +
    3888. Bayes' Theorem and Ridge and Lasso Regression
      3889. +
    3889. Test Function for what happens with OLS, Ridge and Lasso
      3890. +
    3890. Invoking Bayes' theorem
      3891. +
    3891. Ridge and Bayes
      3892. +
    3892. Lasso and Bayes
      3893. +
    3893. Exercise 1: mean values and variances in linear regression
      3894. +
    3894. Exercise 2: Adding Ridge and Lasso Regression
      3895. @@ -297,47 +295,39 @@ MathJax.Hub.Config({

       

       

       

      -

      Still thinking

      +

      What does centering (subtracting the mean values) mean mathematically?

      -

      Keep in mind that when you transform your data set before training a model, the same transformation needs to be done -on your eventual new data set before making a prediction. If we translate this into a Python code, it would could be implemented as follows +

      Let us try to understand what this may imply mathematically when we +subtract the mean values, also known as zero centering. For +simplicity, we will focus on ordinary regression, as done in the above example.

      +

      The cost/loss function for regression is

      +$$ +C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,. +$$ - -
      -
      -
      -
      -
      - 
      #Model training, we compute the mean value of y and X
-y_train_mean = np.mean(y_train)
-X_train_mean = np.mean(X_train,axis=0)
-X_train = X_train - X_train_mean
-y_train = y_train - y_train_mean
+
      Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.
      
 
-# The we fit our model with the training data
-trained_model = some_model.fit(X_train,y_train)
+
      What we have done is to single out the \( \beta_0 \) term in the definition of the mean squared error (MSE).
+The design matrix
+\( X \) does in this case not contain any intercept column.
+When we take the derivative with respect to \( \beta_0 \), we want the derivative to obey
+
      
+$$
+\frac{\partial C}{\partial \beta_j} = 0,
+$$
 
+
      for all \( j \). For \( \beta_0 \) we have
      
 
-#Model prediction, we need also to transform our data set used for the prediction.
-X_test = X_test - X_train_mean #Use mean from training data
-y_pred = trained_model(X_test)
-y_pred = y_pred + y_train_mean
-
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      +$$ +\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right). +$$ + +

      Multiplying away the constant \( 2/n \), we obtain

      +$$ +\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j. +$$

      @@ -365,7 +355,7 @@ y_pred = y_pred 41

    3895. 42
    3896. ...
      3897. -
    3897. 60
      3898. +
    3898. 59
    3899. »
      3900. diff --git a/doc/pub/week36/html/._week36-bs033.html b/doc/pub/week36/html/._week36-bs033.html index e4682d120..ad9dd29d6 100644 --- a/doc/pub/week36/html/._week36-bs033.html +++ b/doc/pub/week36/html/._week36-bs033.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    3900. Plans for week 36
      3901. -
    3901. Thursday September 8
      3902. -
    3902. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3903. -
    3903. Linear Regression and the SVD
      3904. -
    3904. What does it mean?
      3905. -
    3905. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3906. -
    3906. Code for SVD and Inversion of Matrices
      3907. -
    3907. Inverse of Rectangular Matrix
      3908. -
    3908. Ridge and LASSO Regression
      3909. -
    3909. From OLS to Ridge and Lasso
      3910. -
    3910. Deriving the Ridge Regression Equations
      3911. -
    3911. Note on Scikit-Learn
      3912. -
    3912. Comparison with OLS
      3913. -
    3913. SVD analysis
      3914. -
    3914. Interpreting the Ridge results
      3915. -
    3915. More interpretations
      3916. -
    3916. Deriving the Lasso Regression Equations
      3917. -
    3917. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3918. -
    3918. Ridge Regression
      3919. -
    3919. Lasso Regression
      3920. -
    3920. Yet another Example
      3921. -
    3921. The OLS case
      3922. -
    3922. The Ridge case
      3923. -
    3923. Writing the Cost Function
      3924. -
    3924. Lasso case
      3925. -
    3925. The first Case
      3926. -
    3926. Simple code for solving the above problem
      3927. -
    3927. With Lasso Regression
      3928. -
    3928. Another Example, now with a polynomial fit
      3929. -
    3929. To think about, first part
      3930. -
    3930. More thinking
      3931. -
    3931. Still thinking
      3932. -
    3932. What does centering (subtracting the mean values) mean mathematically?
      3933. -
    3933. Further Manipulations
      3934. -
    3934. Wrapping it up
      3935. -
    3935. Linear Regression code, Intercept handling first
      3936. -
    3936. Code Examples
      3937. -
    3937. Taking out the mean
      3938. -
    3938. Friday September 9
      3939. -
    3939. Linking the regression analysis with a statistical interpretation
      3940. -
    3940. Assumptions made
      3941. -
    3941. Expectation value and variance
      3942. -
    3942. Expectation value and variance for \( \boldsymbol{\beta} \)
      3943. -
    3943. Deriving OLS from a probability distribution
      3944. -
    3944. Independent and Identically Distrubuted (iid)
      3945. -
    3945. Maximum Likelihood Estimation (MLE)
      3946. -
    3946. A new Cost Function
      3947. -
    3947. More basic Statistics and Bayes' theorem
      3948. -
    3948. Marginal Probability
      3949. -
    3949. Conditional Probability
      3950. -
    3950. Bayes' Theorem
      3951. -
    3951. Interpretations of Bayes' Theorem
      3952. -
    3952. Example of Usage of Bayes' theorem
      3953. -
    3953. Doing it correctly
      3954. -
    3954. Bayes' Theorem and Ridge and Lasso Regression
      3955. -
    3955. Test Function for what happens with OLS, Ridge and Lasso
      3956. -
    3956. Invoking Bayes' theorem
      3957. -
    3957. Ridge and Bayes
      3958. -
    3958. Lasso and Bayes
      3959. -
    3959. Exercise 1: mean values and variances in linear regression
      3960. -
    3960. Exercise 2: Adding Ridge and Lasso Regression
      3961. +
    3961. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      3962. +
    3962. Linear Regression and the SVD
      3963. +
    3963. What does it mean?
      3964. +
    3964. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      3965. +
    3965. Code for SVD and Inversion of Matrices
      3966. +
    3966. Inverse of Rectangular Matrix
      3967. +
    3967. Ridge and LASSO Regression
      3968. +
    3968. From OLS to Ridge and Lasso
      3969. +
    3969. Deriving the Ridge Regression Equations
      3970. +
    3970. Note on Scikit-Learn
      3971. +
    3971. Comparison with OLS
      3972. +
    3972. SVD analysis
      3973. +
    3973. Interpreting the Ridge results
      3974. +
    3974. More interpretations
      3975. +
    3975. Deriving the Lasso Regression Equations
      3976. +
    3976. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      3977. +
    3977. Ridge Regression
      3978. +
    3978. Lasso Regression
      3979. +
    3979. Yet another Example
      3980. +
    3980. The OLS case
      3981. +
    3981. The Ridge case
      3982. +
    3982. Writing the Cost Function
      3983. +
    3983. Lasso case
      3984. +
    3984. The first Case
      3985. +
    3985. Simple code for solving the above problem
      3986. +
    3986. With Lasso Regression
      3987. +
    3987. Another Example, now with a polynomial fit
      3988. +
    3988. To think about, first part
      3989. +
    3989. More thinking
      3990. +
    3990. Still thinking
      3991. +
    3991. What does centering (subtracting the mean values) mean mathematically?
      3992. +
    3992. Further Manipulations
      3993. +
    3993. Wrapping it up
      3994. +
    3994. Linear Regression code, Intercept handling first
      3995. +
    3995. Code Examples
      3996. +
    3996. Taking out the mean
      3997. +
    3997. Friday September 9
      3998. +
    3998. Linking the regression analysis with a statistical interpretation
      3999. +
    3999. Assumptions made
      4000. +
    4000. Expectation value and variance
      4001. +
    4001. Expectation value and variance for \( \boldsymbol{\beta} \)
      4002. +
    4002. Deriving OLS from a probability distribution
      4003. +
    4003. Independent and Identically Distrubuted (iid)
      4004. +
    4004. Maximum Likelihood Estimation (MLE)
      4005. +
    4005. A new Cost Function
      4006. +
    4006. More basic Statistics and Bayes' theorem
      4007. +
    4007. Marginal Probability
      4008. +
    4008. Conditional Probability
      4009. +
    4009. Bayes' Theorem
      4010. +
    4010. Interpretations of Bayes' Theorem
      4011. +
    4011. Example of Usage of Bayes' theorem
      4012. +
    4012. Doing it correctly
      4013. +
    4013. Bayes' Theorem and Ridge and Lasso Regression
      4014. +
    4014. Test Function for what happens with OLS, Ridge and Lasso
      4015. +
    4015. Invoking Bayes' theorem
      4016. +
    4016. Ridge and Bayes
      4017. +
    4017. Lasso and Bayes
      4018. +
    4018. Exercise 1: mean values and variances in linear regression
      4019. +
    4019. Exercise 2: Adding Ridge and Lasso Regression
      4020. @@ -297,38 +295,43 @@ MathJax.Hub.Config({

       

       

       

      -

      What does centering (subtracting the mean values) mean mathematically?

      +

      Further Manipulations

      -

      Let us try to understand what this may imply mathematically when we -subtract the mean values, also known as zero centering. For -simplicity, we will focus on ordinary regression, as done in the above example. -

      - -

      The cost/loss function for regression is

      -$$ -C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,. -$$ - -

      Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.

      - -

      What we have done is to single out the \( \beta_0 \) term in the definition of the mean squared error (MSE). -The design matrix -\( X \) does in this case not contain any intercept column. -When we take the derivative with respect to \( \beta_0 \), we want the derivative to obey +

      Let us special first to the case where we have only two parameters \( \beta_0 \) and \( \beta_1 \). +Our result for \( \beta_0 \) simplifies then to

      $$ -\frac{\partial C}{\partial \beta_j} = 0, +n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1. $$ -

      for all \( j \). For \( \beta_0 \) we have

      - +

      We obtain then

      $$ -\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right). +\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. $$ -

      Multiplying away the constant \( 2/n \), we obtain

      +

      If we define

      $$ -\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j. +\mu_1=\frac{1}{n}\sum_{i=0}^{n-1} (X_{i1}, +$$ + +

      and if we define the mean value of the outputs as

      +$$ +\mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i, +$$ + +

      we have

      +$$ +\beta_0 = \mu_y - \beta_1\mu_{1}. +$$ + +

      In the general case, that is we have more parameters than \( \beta_0 \) and \( \beta_1 \), we have

      +$$ +\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j. +$$ + +

      Replacing \( y_i \) with \( y_i - y_i - \overline{\boldsymbol{y}} \) and centering also our design matrix results in a cost function (in vector-matrix disguise)

      +$$ +C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}). $$ @@ -357,7 +360,7 @@ $$
    4020. 42
    4021. 43
    4022. ...
      4023. -
    4023. 60
      4024. +
    4024. 59
    4025. »
      4026. diff --git a/doc/pub/week36/html/._week36-bs034.html b/doc/pub/week36/html/._week36-bs034.html index aa618dc1b..13a93dad3 100644 --- a/doc/pub/week36/html/._week36-bs034.html +++ b/doc/pub/week36/html/._week36-bs034.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    4026. Plans for week 36
      4027. -
    4027. Thursday September 8
      4028. -
    4028. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4029. -
    4029. Linear Regression and the SVD
      4030. -
    4030. What does it mean?
      4031. -
    4031. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4032. -
    4032. Code for SVD and Inversion of Matrices
      4033. -
    4033. Inverse of Rectangular Matrix
      4034. -
    4034. Ridge and LASSO Regression
      4035. -
    4035. From OLS to Ridge and Lasso
      4036. -
    4036. Deriving the Ridge Regression Equations
      4037. -
    4037. Note on Scikit-Learn
      4038. -
    4038. Comparison with OLS
      4039. -
    4039. SVD analysis
      4040. -
    4040. Interpreting the Ridge results
      4041. -
    4041. More interpretations
      4042. -
    4042. Deriving the Lasso Regression Equations
      4043. -
    4043. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4044. -
    4044. Ridge Regression
      4045. -
    4045. Lasso Regression
      4046. -
    4046. Yet another Example
      4047. -
    4047. The OLS case
      4048. -
    4048. The Ridge case
      4049. -
    4049. Writing the Cost Function
      4050. -
    4050. Lasso case
      4051. -
    4051. The first Case
      4052. -
    4052. Simple code for solving the above problem
      4053. -
    4053. With Lasso Regression
      4054. -
    4054. Another Example, now with a polynomial fit
      4055. -
    4055. To think about, first part
      4056. -
    4056. More thinking
      4057. -
    4057. Still thinking
      4058. -
    4058. What does centering (subtracting the mean values) mean mathematically?
      4059. -
    4059. Further Manipulations
      4060. -
    4060. Wrapping it up
      4061. -
    4061. Linear Regression code, Intercept handling first
      4062. -
    4062. Code Examples
      4063. -
    4063. Taking out the mean
      4064. -
    4064. Friday September 9
      4065. -
    4065. Linking the regression analysis with a statistical interpretation
      4066. -
    4066. Assumptions made
      4067. -
    4067. Expectation value and variance
      4068. -
    4068. Expectation value and variance for \( \boldsymbol{\beta} \)
      4069. -
    4069. Deriving OLS from a probability distribution
      4070. -
    4070. Independent and Identically Distrubuted (iid)
      4071. -
    4071. Maximum Likelihood Estimation (MLE)
      4072. -
    4072. A new Cost Function
      4073. -
    4073. More basic Statistics and Bayes' theorem
      4074. -
    4074. Marginal Probability
      4075. -
    4075. Conditional Probability
      4076. -
    4076. Bayes' Theorem
      4077. -
    4077. Interpretations of Bayes' Theorem
      4078. -
    4078. Example of Usage of Bayes' theorem
      4079. -
    4079. Doing it correctly
      4080. -
    4080. Bayes' Theorem and Ridge and Lasso Regression
      4081. -
    4081. Test Function for what happens with OLS, Ridge and Lasso
      4082. -
    4082. Invoking Bayes' theorem
      4083. -
    4083. Ridge and Bayes
      4084. -
    4084. Lasso and Bayes
      4085. -
    4085. Exercise 1: mean values and variances in linear regression
      4086. -
    4086. Exercise 2: Adding Ridge and Lasso Regression
      4087. +
    4087. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4088. +
    4088. Linear Regression and the SVD
      4089. +
    4089. What does it mean?
      4090. +
    4090. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4091. +
    4091. Code for SVD and Inversion of Matrices
      4092. +
    4092. Inverse of Rectangular Matrix
      4093. +
    4093. Ridge and LASSO Regression
      4094. +
    4094. From OLS to Ridge and Lasso
      4095. +
    4095. Deriving the Ridge Regression Equations
      4096. +
    4096. Note on Scikit-Learn
      4097. +
    4097. Comparison with OLS
      4098. +
    4098. SVD analysis
      4099. +
    4099. Interpreting the Ridge results
      4100. +
    4100. More interpretations
      4101. +
    4101. Deriving the Lasso Regression Equations
      4102. +
    4102. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4103. +
    4103. Ridge Regression
      4104. +
    4104. Lasso Regression
      4105. +
    4105. Yet another Example
      4106. +
    4106. The OLS case
      4107. +
    4107. The Ridge case
      4108. +
    4108. Writing the Cost Function
      4109. +
    4109. Lasso case
      4110. +
    4110. The first Case
      4111. +
    4111. Simple code for solving the above problem
      4112. +
    4112. With Lasso Regression
      4113. +
    4113. Another Example, now with a polynomial fit
      4114. +
    4114. To think about, first part
      4115. +
    4115. More thinking
      4116. +
    4116. Still thinking
      4117. +
    4117. What does centering (subtracting the mean values) mean mathematically?
      4118. +
    4118. Further Manipulations
      4119. +
    4119. Wrapping it up
      4120. +
    4120. Linear Regression code, Intercept handling first
      4121. +
    4121. Code Examples
      4122. +
    4122. Taking out the mean
      4123. +
    4123. Friday September 9
      4124. +
    4124. Linking the regression analysis with a statistical interpretation
      4125. +
    4125. Assumptions made
      4126. +
    4126. Expectation value and variance
      4127. +
    4127. Expectation value and variance for \( \boldsymbol{\beta} \)
      4128. +
    4128. Deriving OLS from a probability distribution
      4129. +
    4129. Independent and Identically Distrubuted (iid)
      4130. +
    4130. Maximum Likelihood Estimation (MLE)
      4131. +
    4131. A new Cost Function
      4132. +
    4132. More basic Statistics and Bayes' theorem
      4133. +
    4133. Marginal Probability
      4134. +
    4134. Conditional Probability
      4135. +
    4135. Bayes' Theorem
      4136. +
    4136. Interpretations of Bayes' Theorem
      4137. +
    4137. Example of Usage of Bayes' theorem
      4138. +
    4138. Doing it correctly
      4139. +
    4139. Bayes' Theorem and Ridge and Lasso Regression
      4140. +
    4140. Test Function for what happens with OLS, Ridge and Lasso
      4141. +
    4141. Invoking Bayes' theorem
      4142. +
    4142. Ridge and Bayes
      4143. +
    4143. Lasso and Bayes
      4144. +
    4144. Exercise 1: mean values and variances in linear regression
      4145. +
    4145. Exercise 2: Adding Ridge and Lasso Regression
      4146. @@ -297,45 +295,24 @@ MathJax.Hub.Config({

       

       

       

      -

      Further Manipulations

      +

      Wrapping it up

      -

      Let us special first to the case where we have only two parameters \( \beta_0 \) and \( \beta_1 \). -Our result for \( \beta_0 \) simplifies then to +

      If we minimize with respect to \( \boldsymbol{\beta} \) we have then

      + +$$ +\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, +$$ + +

      where \( \boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}} \) +and \( \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj} \).

      + +

      For Ridge regression we need to add \( \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} \) to the cost function and get then

      $$ -n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1. -$$ - -

      We obtain then

      -$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. -$$ - -

      If we define

      -$$ -\mu_1=\frac{1}{n}\sum_{i=0}^{n-1} (X_{i1}, -$$ - -

      and if we define the mean value of the outputs as

      -$$ -\mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i, -$$ - -

      we have

      -$$ -\beta_0 = \mu_y - \beta_1\mu_{1}. -$$ - -

      In the general case, that is we have more parameters than \( \beta_0 \) and \( \beta_1 \), we have

      -$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j. -$$ - -

      Replacing \( y_i \) with \( y_i - y_i - \overline{\boldsymbol{y}} \) and centering also our design matrix results in a cost function (in vector-matrix disguise)

      -$$ -C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}). +\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. $$ +

      What does this mean? And why do we insist on all this? Let us look at some examples.

      @@ -362,7 +339,7 @@ $$

    4146. 43
    4147. 44
    4148. ...
      4149. -
    4149. 60
      4150. +
    4150. 59
    4151. »
      4152. diff --git a/doc/pub/week36/html/._week36-bs035.html b/doc/pub/week36/html/._week36-bs035.html index f1f7af7cd..64ff998bf 100644 --- a/doc/pub/week36/html/._week36-bs035.html +++ b/doc/pub/week36/html/._week36-bs035.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    4152. Plans for week 36
      4153. -
    4153. Thursday September 8
      4154. -
    4154. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4155. -
    4155. Linear Regression and the SVD
      4156. -
    4156. What does it mean?
      4157. -
    4157. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4158. -
    4158. Code for SVD and Inversion of Matrices
      4159. -
    4159. Inverse of Rectangular Matrix
      4160. -
    4160. Ridge and LASSO Regression
      4161. -
    4161. From OLS to Ridge and Lasso
      4162. -
    4162. Deriving the Ridge Regression Equations
      4163. -
    4163. Note on Scikit-Learn
      4164. -
    4164. Comparison with OLS
      4165. -
    4165. SVD analysis
      4166. -
    4166. Interpreting the Ridge results
      4167. -
    4167. More interpretations
      4168. -
    4168. Deriving the Lasso Regression Equations
      4169. -
    4169. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4170. -
    4170. Ridge Regression
      4171. -
    4171. Lasso Regression
      4172. -
    4172. Yet another Example
      4173. -
    4173. The OLS case
      4174. -
    4174. The Ridge case
      4175. -
    4175. Writing the Cost Function
      4176. -
    4176. Lasso case
      4177. -
    4177. The first Case
      4178. -
    4178. Simple code for solving the above problem
      4179. -
    4179. With Lasso Regression
      4180. -
    4180. Another Example, now with a polynomial fit
      4181. -
    4181. To think about, first part
      4182. -
    4182. More thinking
      4183. -
    4183. Still thinking
      4184. -
    4184. What does centering (subtracting the mean values) mean mathematically?
      4185. -
    4185. Further Manipulations
      4186. -
    4186. Wrapping it up
      4187. -
    4187. Linear Regression code, Intercept handling first
      4188. -
    4188. Code Examples
      4189. -
    4189. Taking out the mean
      4190. -
    4190. Friday September 9
      4191. -
    4191. Linking the regression analysis with a statistical interpretation
      4192. -
    4192. Assumptions made
      4193. -
    4193. Expectation value and variance
      4194. -
    4194. Expectation value and variance for \( \boldsymbol{\beta} \)
      4195. -
    4195. Deriving OLS from a probability distribution
      4196. -
    4196. Independent and Identically Distrubuted (iid)
      4197. -
    4197. Maximum Likelihood Estimation (MLE)
      4198. -
    4198. A new Cost Function
      4199. -
    4199. More basic Statistics and Bayes' theorem
      4200. -
    4200. Marginal Probability
      4201. -
    4201. Conditional Probability
      4202. -
    4202. Bayes' Theorem
      4203. -
    4203. Interpretations of Bayes' Theorem
      4204. -
    4204. Example of Usage of Bayes' theorem
      4205. -
    4205. Doing it correctly
      4206. -
    4206. Bayes' Theorem and Ridge and Lasso Regression
      4207. -
    4207. Test Function for what happens with OLS, Ridge and Lasso
      4208. -
    4208. Invoking Bayes' theorem
      4209. -
    4209. Ridge and Bayes
      4210. -
    4210. Lasso and Bayes
      4211. -
    4211. Exercise 1: mean values and variances in linear regression
      4212. -
    4212. Exercise 2: Adding Ridge and Lasso Regression
      4213. +
    4213. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4214. +
    4214. Linear Regression and the SVD
      4215. +
    4215. What does it mean?
      4216. +
    4216. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4217. +
    4217. Code for SVD and Inversion of Matrices
      4218. +
    4218. Inverse of Rectangular Matrix
      4219. +
    4219. Ridge and LASSO Regression
      4220. +
    4220. From OLS to Ridge and Lasso
      4221. +
    4221. Deriving the Ridge Regression Equations
      4222. +
    4222. Note on Scikit-Learn
      4223. +
    4223. Comparison with OLS
      4224. +
    4224. SVD analysis
      4225. +
    4225. Interpreting the Ridge results
      4226. +
    4226. More interpretations
      4227. +
    4227. Deriving the Lasso Regression Equations
      4228. +
    4228. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4229. +
    4229. Ridge Regression
      4230. +
    4230. Lasso Regression
      4231. +
    4231. Yet another Example
      4232. +
    4232. The OLS case
      4233. +
    4233. The Ridge case
      4234. +
    4234. Writing the Cost Function
      4235. +
    4235. Lasso case
      4236. +
    4236. The first Case
      4237. +
    4237. Simple code for solving the above problem
      4238. +
    4238. With Lasso Regression
      4239. +
    4239. Another Example, now with a polynomial fit
      4240. +
    4240. To think about, first part
      4241. +
    4241. More thinking
      4242. +
    4242. Still thinking
      4243. +
    4243. What does centering (subtracting the mean values) mean mathematically?
      4244. +
    4244. Further Manipulations
      4245. +
    4245. Wrapping it up
      4246. +
    4246. Linear Regression code, Intercept handling first
      4247. +
    4247. Code Examples
      4248. +
    4248. Taking out the mean
      4249. +
    4249. Friday September 9
      4250. +
    4250. Linking the regression analysis with a statistical interpretation
      4251. +
    4251. Assumptions made
      4252. +
    4252. Expectation value and variance
      4253. +
    4253. Expectation value and variance for \( \boldsymbol{\beta} \)
      4254. +
    4254. Deriving OLS from a probability distribution
      4255. +
    4255. Independent and Identically Distrubuted (iid)
      4256. +
    4256. Maximum Likelihood Estimation (MLE)
      4257. +
    4257. A new Cost Function
      4258. +
    4258. More basic Statistics and Bayes' theorem
      4259. +
    4259. Marginal Probability
      4260. +
    4260. Conditional Probability
      4261. +
    4261. Bayes' Theorem
      4262. +
    4262. Interpretations of Bayes' Theorem
      4263. +
    4263. Example of Usage of Bayes' theorem
      4264. +
    4264. Doing it correctly
      4265. +
    4265. Bayes' Theorem and Ridge and Lasso Regression
      4266. +
    4266. Test Function for what happens with OLS, Ridge and Lasso
      4267. +
    4267. Invoking Bayes' theorem
      4268. +
    4268. Ridge and Bayes
      4269. +
    4269. Lasso and Bayes
      4270. +
    4270. Exercise 1: mean values and variances in linear regression
      4271. +
    4271. Exercise 2: Adding Ridge and Lasso Regression
      4272. @@ -297,24 +295,149 @@ MathJax.Hub.Config({

       

       

       

      -

      Wrapping it up

      +

      Linear Regression code, Intercept handling first

      -

      If we minimize with respect to \( \boldsymbol{\beta} \) we have then

      - -$$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, -$$ - -

      where \( \boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}} \) -and \( \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj} \). +

      This code shows a simple first-order fit to a data set using the above transformed data, where we consider the role of the intercept first, by either excluding it or including it (code example thanks to Øyvind Sigmundson Schøyen). Here our scaling of the data is done by subtracting the mean values only. +Note also that we do not split the data into training and test.

      -

      For Ridge regression we need to add \( \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} \) to the cost function and get then

      + + +
      +
      +
      +
      +
      + 
      import numpy as np
+import matplotlib.pyplot as plt
+
+from sklearn.linear_model import LinearRegression
+
+
+np.random.seed(2021)
+
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+
+def fit_beta(X, y):
+    return np.linalg.pinv(X.T @ X) @ X.T @ y
+
+
+true_beta = [2, 0.5, 3.7]
+
+x = np.linspace(0, 1, 11)
+y = np.sum(
+    np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0
+) + 0.1 * np.random.normal(size=len(x))
+
+degree = 3
+X = np.zeros((len(x), degree))
+
+# Include the intercept in the design matrix
+for p in range(degree):
+    X[:, p] = x ** p
+
+beta = fit_beta(X, y)
+
+# Intercept is included in the design matrix
+skl = LinearRegression(fit_intercept=False).fit(X, y)
+
+print(f"True beta: {true_beta}")
+print(f"Fitted beta: {beta}")
+print(f"Sklearn fitted beta: {skl.coef_}")
+ypredictOwn = X @ beta
+ypredictSKL = skl.predict(X)
+print(f"MSE with intercept column")
+print(MSE(y,ypredictOwn))
+print(f"MSE with intercept column from SKL")
+print(MSE(y,ypredictSKL))
+
+
+plt.figure()
+plt.scatter(x, y, label="Data")
+plt.plot(x, X @ beta, label="Fit")
+plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)")
+
+
+# Do not include the intercept in the design matrix
+X = np.zeros((len(x), degree - 1))
+
+for p in range(degree - 1):
+    X[:, p] = x ** (p + 1)
+
+# Intercept is not included in the design matrix
+skl = LinearRegression(fit_intercept=True).fit(X, y)
+
+# Use centered values for X and y when computing coefficients
+y_offset = np.average(y, axis=0)
+X_offset = np.average(X, axis=0)
+
+beta = fit_beta(X - X_offset, y - y_offset)
+intercept = np.mean(y_offset - X_offset @ beta)
+
+print(f"Manual intercept: {intercept}")
+print(f"Fitted beta (wiothout intercept): {beta}")
+print(f"Sklearn intercept: {skl.intercept_}")
+print(f"Sklearn fitted beta (without intercept): {skl.coef_}")
+ypredictOwn = X @ beta
+ypredictSKL = skl.predict(X)
+print(f"MSE with Manual intercept")
+print(MSE(y,ypredictOwn+intercept))
+print(f"MSE with Sklearn intercept")
+print(MSE(y,ypredictSKL))
+
+plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)")
+plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)")
+plt.grid()
+plt.legend()
+
+plt.show()
+
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      + +

      The intercept is the value of our output/target variable +when all our features are zero and our function crosses the \( y \)-axis (for a one-dimensional case). +

      + +

      Printing the MSE, we see first that both methods give the same MSE, as +they should. However, when we move to for example Ridge regression, +the way we treat the intercept may give a larger or smaller MSE, +meaning that the MSE can be penalized by the value of the +intercept. Not including the intercept in the fit, means that the +regularization term does not include \( \beta_0 \). For different values +of \( \lambda \), this may lead to differeing MSE values. +

      + +

      To remind the reader, the regularization term, with the intercept in Ridge regression is given by

      $$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. +\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2, $$ -

      What does this mean? And why do we insist on all this? Let us look at some examples.

      +

      but when we take out the intercept, this equation becomes

      +$$ +\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2. +$$ + +

      For Lasso regression we have

      +$$ +\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. +$$ + +

      It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get a MSE which then contains the intercept.

      @@ -341,7 +464,7 @@ $$

    4272. 44
    4273. 45
    4274. ...
      4275. -
    4275. 60
      4276. +
    4276. 59
    4277. »
      4278. diff --git a/doc/pub/week36/html/._week36-bs036.html b/doc/pub/week36/html/._week36-bs036.html index d3056dd33..7cda3efb7 100644 --- a/doc/pub/week36/html/._week36-bs036.html +++ b/doc/pub/week36/html/._week36-bs036.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    4278. Plans for week 36
      4279. -
    4279. Thursday September 8
      4280. -
    4280. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4281. -
    4281. Linear Regression and the SVD
      4282. -
    4282. What does it mean?
      4283. -
    4283. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4284. -
    4284. Code for SVD and Inversion of Matrices
      4285. -
    4285. Inverse of Rectangular Matrix
      4286. -
    4286. Ridge and LASSO Regression
      4287. -
    4287. From OLS to Ridge and Lasso
      4288. -
    4288. Deriving the Ridge Regression Equations
      4289. -
    4289. Note on Scikit-Learn
      4290. -
    4290. Comparison with OLS
      4291. -
    4291. SVD analysis
      4292. -
    4292. Interpreting the Ridge results
      4293. -
    4293. More interpretations
      4294. -
    4294. Deriving the Lasso Regression Equations
      4295. -
    4295. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4296. -
    4296. Ridge Regression
      4297. -
    4297. Lasso Regression
      4298. -
    4298. Yet another Example
      4299. -
    4299. The OLS case
      4300. -
    4300. The Ridge case
      4301. -
    4301. Writing the Cost Function
      4302. -
    4302. Lasso case
      4303. -
    4303. The first Case
      4304. -
    4304. Simple code for solving the above problem
      4305. -
    4305. With Lasso Regression
      4306. -
    4306. Another Example, now with a polynomial fit
      4307. -
    4307. To think about, first part
      4308. -
    4308. More thinking
      4309. -
    4309. Still thinking
      4310. -
    4310. What does centering (subtracting the mean values) mean mathematically?
      4311. -
    4311. Further Manipulations
      4312. -
    4312. Wrapping it up
      4313. -
    4313. Linear Regression code, Intercept handling first
      4314. -
    4314. Code Examples
      4315. -
    4315. Taking out the mean
      4316. -
    4316. Friday September 9
      4317. -
    4317. Linking the regression analysis with a statistical interpretation
      4318. -
    4318. Assumptions made
      4319. -
    4319. Expectation value and variance
      4320. -
    4320. Expectation value and variance for \( \boldsymbol{\beta} \)
      4321. -
    4321. Deriving OLS from a probability distribution
      4322. -
    4322. Independent and Identically Distrubuted (iid)
      4323. -
    4323. Maximum Likelihood Estimation (MLE)
      4324. -
    4324. A new Cost Function
      4325. -
    4325. More basic Statistics and Bayes' theorem
      4326. -
    4326. Marginal Probability
      4327. -
    4327. Conditional Probability
      4328. -
    4328. Bayes' Theorem
      4329. -
    4329. Interpretations of Bayes' Theorem
      4330. -
    4330. Example of Usage of Bayes' theorem
      4331. -
    4331. Doing it correctly
      4332. -
    4332. Bayes' Theorem and Ridge and Lasso Regression
      4333. -
    4333. Test Function for what happens with OLS, Ridge and Lasso
      4334. -
    4334. Invoking Bayes' theorem
      4335. -
    4335. Ridge and Bayes
      4336. -
    4336. Lasso and Bayes
      4337. -
    4337. Exercise 1: mean values and variances in linear regression
      4338. -
    4338. Exercise 2: Adding Ridge and Lasso Regression
      4339. +
    4339. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4340. +
    4340. Linear Regression and the SVD
      4341. +
    4341. What does it mean?
      4342. +
    4342. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4343. +
    4343. Code for SVD and Inversion of Matrices
      4344. +
    4344. Inverse of Rectangular Matrix
      4345. +
    4345. Ridge and LASSO Regression
      4346. +
    4346. From OLS to Ridge and Lasso
      4347. +
    4347. Deriving the Ridge Regression Equations
      4348. +
    4348. Note on Scikit-Learn
      4349. +
    4349. Comparison with OLS
      4350. +
    4350. SVD analysis
      4351. +
    4351. Interpreting the Ridge results
      4352. +
    4352. More interpretations
      4353. +
    4353. Deriving the Lasso Regression Equations
      4354. +
    4354. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4355. +
    4355. Ridge Regression
      4356. +
    4356. Lasso Regression
      4357. +
    4357. Yet another Example
      4358. +
    4358. The OLS case
      4359. +
    4359. The Ridge case
      4360. +
    4360. Writing the Cost Function
      4361. +
    4361. Lasso case
      4362. +
    4362. The first Case
      4363. +
    4363. Simple code for solving the above problem
      4364. +
    4364. With Lasso Regression
      4365. +
    4365. Another Example, now with a polynomial fit
      4366. +
    4366. To think about, first part
      4367. +
    4367. More thinking
      4368. +
    4368. Still thinking
      4369. +
    4369. What does centering (subtracting the mean values) mean mathematically?
      4370. +
    4370. Further Manipulations
      4371. +
    4371. Wrapping it up
      4372. +
    4372. Linear Regression code, Intercept handling first
      4373. +
    4373. Code Examples
      4374. +
    4374. Taking out the mean
      4375. +
    4375. Friday September 9
      4376. +
    4376. Linking the regression analysis with a statistical interpretation
      4377. +
    4377. Assumptions made
      4378. +
    4378. Expectation value and variance
      4379. +
    4379. Expectation value and variance for \( \boldsymbol{\beta} \)
      4380. +
    4380. Deriving OLS from a probability distribution
      4381. +
    4381. Independent and Identically Distrubuted (iid)
      4382. +
    4382. Maximum Likelihood Estimation (MLE)
      4383. +
    4383. A new Cost Function
      4384. +
    4384. More basic Statistics and Bayes' theorem
      4385. +
    4385. Marginal Probability
      4386. +
    4386. Conditional Probability
      4387. +
    4387. Bayes' Theorem
      4388. +
    4388. Interpretations of Bayes' Theorem
      4389. +
    4389. Example of Usage of Bayes' theorem
      4390. +
    4390. Doing it correctly
      4391. +
    4391. Bayes' Theorem and Ridge and Lasso Regression
      4392. +
    4392. Test Function for what happens with OLS, Ridge and Lasso
      4393. +
    4393. Invoking Bayes' theorem
      4394. +
    4394. Ridge and Bayes
      4395. +
    4395. Lasso and Bayes
      4396. +
    4396. Exercise 1: mean values and variances in linear regression
      4397. +
    4397. Exercise 2: Adding Ridge and Lasso Regression
      4398. @@ -297,11 +295,9 @@ MathJax.Hub.Config({

       

       

       

      -

      Linear Regression code, Intercept handling first

      +

      Code Examples

      -

      This code shows a simple first-order fit to a data set using the above transformed data, where we consider the role of the intercept first, by either excluding it or including it (code example thanks to Øyvind Sigmundson Schøyen). Here our scaling of the data is done by subtracting the mean values only. -Note also that we do not split the data into training and test. -

      +

      Armed with this wisdom, we attempt first to simply set the intercept equal to False in our implementation of Ridge regression for our well-known vanilla data set.

      @@ -311,90 +307,69 @@ Note also that we do not split the data into training and test. 
      import numpy as np
+import pandas as pd
 import matplotlib.pyplot as plt
-
-from sklearn.linear_model import LinearRegression
-
-
-np.random.seed(2021)
+from sklearn.model_selection import train_test_split
+from sklearn import linear_model
 
 def MSE(y_data,y_model):
     n = np.size(y_model)
     return np.sum((y_data-y_model)**2)/n
 
 
-def fit_beta(X, y):
-    return np.linalg.pinv(X.T @ X) @ X.T @ y
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
 
+n = 100
+x = np.random.rand(n)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)
 
-true_beta = [2, 0.5, 3.7]
-
-x = np.linspace(0, 1, 11)
-y = np.sum(
-    np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0
-) + 0.1 * np.random.normal(size=len(x))
-
-degree = 3
-X = np.zeros((len(x), degree))
-
-# Include the intercept in the design matrix
-for p in range(degree):
-    X[:, p] = x ** p
-
-beta = fit_beta(X, y)
-
-# Intercept is included in the design matrix
-skl = LinearRegression(fit_intercept=False).fit(X, y)
-
-print(f"True beta: {true_beta}")
-print(f"Fitted beta: {beta}")
-print(f"Sklearn fitted beta: {skl.coef_}")
-ypredictOwn = X @ beta
-ypredictSKL = skl.predict(X)
-print(f"MSE with intercept column")
-print(MSE(y,ypredictOwn))
-print(f"MSE with intercept column from SKL")
-print(MSE(y,ypredictSKL))
+Maxpolydegree = 20
+X = np.zeros((n,Maxpolydegree))
+#We include explicitely the intercept column
+for degree in range(Maxpolydegree):
+    X[:,degree] = x**degree
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
 
+p = Maxpolydegree
+I = np.eye(p,p)
+# Decide which values of lambda to use
+nlambdas = 6
+MSEOwnRidgePredict = np.zeros(nlambdas)
+MSERidgePredict = np.zeros(nlambdas)
+lambdas = np.logspace(-4, 2, nlambdas)
+for i in range(nlambdas):
+    lmb = lambdas[i]
+    OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
+    # Note: we include the intercept column and no scaling
+    RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
+    RegRidge.fit(X_train,y_train)
+    # and then make the prediction
+    ytildeOwnRidge = X_train @ OwnRidgeBeta
+    ypredictOwnRidge = X_test @ OwnRidgeBeta
+    ytildeRidge = RegRidge.predict(X_train)
+    ypredictRidge = RegRidge.predict(X_test)
+    MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
+    MSERidgePredict[i] = MSE(y_test,ypredictRidge)
+    print("Beta values for own Ridge implementation")
+    print(OwnRidgeBeta)
+    print("Beta values for Scikit-Learn Ridge implementation")
+    print(RegRidge.coef_)
+    print("MSE values for own Ridge implementation")
+    print(MSEOwnRidgePredict[i])
+    print("MSE values for Scikit-Learn Ridge implementation")
+    print(MSERidgePredict[i])
 
+# Now plot the results
 plt.figure()
-plt.scatter(x, y, label="Data")
-plt.plot(x, X @ beta, label="Fit")
-plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)")
+plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')
+plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')
 
-
-# Do not include the intercept in the design matrix
-X = np.zeros((len(x), degree - 1))
-
-for p in range(degree - 1):
-    X[:, p] = x ** (p + 1)
-
-# Intercept is not included in the design matrix
-skl = LinearRegression(fit_intercept=True).fit(X, y)
-
-# Use centered values for X and y when computing coefficients
-y_offset = np.average(y, axis=0)
-X_offset = np.average(X, axis=0)
-
-beta = fit_beta(X - X_offset, y - y_offset)
-intercept = np.mean(y_offset - X_offset @ beta)
-
-print(f"Manual intercept: {intercept}")
-print(f"Fitted beta (wiothout intercept): {beta}")
-print(f"Sklearn intercept: {skl.intercept_}")
-print(f"Sklearn fitted beta (without intercept): {skl.coef_}")
-ypredictOwn = X @ beta
-ypredictSKL = skl.predict(X)
-print(f"MSE with Manual intercept")
-print(MSE(y,ypredictOwn+intercept))
-print(f"MSE with Sklearn intercept")
-print(MSE(y,ypredictSKL))
-
-plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)")
-plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)")
-plt.grid()
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
 plt.legend()
-
 plt.show()
      @@ -411,36 +386,12 @@ plt.show()
      -

      The intercept is the value of our output/target variable -when all our features are zero and our function crosses the \( y \)-axis (for a one-dimensional case). +

      The results here agree when we force Scikit-Learn's Ridge function to include the first column in our design matrix. +We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix. +What happens if we do not include the intercept in our fit? +Let us see how we can change this code by zero centering (thanks to Stian Bilek for inpouts here).

      -

      Printing the MSE, we see first that both methods give the same MSE, as -they should. However, when we move to for example Ridge regression, -the way we treat the intercept may give a larger or smaller MSE, -meaning that the MSE can be penalized by the value of the -intercept. Not including the intercept in the fit, means that the -regularization term does not include \( \beta_0 \). For different values -of \( \lambda \), this may lead to differeing MSE values. -

      - -

      To remind the reader, the regularization term, with the intercept in Ridge regression is given by

      -$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2, -$$ - -

      but when we take out the intercept, this equation becomes

      -$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2. -$$ - -

      For Lasso regression we have

      -$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. -$$ - -

      It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get a MSE which then contains the intercept.

      -

      diff --git a/doc/pub/week36/html/._week36-bs037.html b/doc/pub/week36/html/._week36-bs037.html index 2ced7b245..cfcab41c0 100644 --- a/doc/pub/week36/html/._week36-bs037.html +++ b/doc/pub/week36/html/._week36-bs037.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    4398. Plans for week 36
      4399. -
    4399. Thursday September 8
      4400. -
    4400. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4401. -
    4401. Linear Regression and the SVD
      4402. -
    4402. What does it mean?
      4403. -
    4403. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4404. -
    4404. Code for SVD and Inversion of Matrices
      4405. -
    4405. Inverse of Rectangular Matrix
      4406. -
    4406. Ridge and LASSO Regression
      4407. -
    4407. From OLS to Ridge and Lasso
      4408. -
    4408. Deriving the Ridge Regression Equations
      4409. -
    4409. Note on Scikit-Learn
      4410. -
    4410. Comparison with OLS
      4411. -
    4411. SVD analysis
      4412. -
    4412. Interpreting the Ridge results
      4413. -
    4413. More interpretations
      4414. -
    4414. Deriving the Lasso Regression Equations
      4415. -
    4415. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4416. -
    4416. Ridge Regression
      4417. -
    4417. Lasso Regression
      4418. -
    4418. Yet another Example
      4419. -
    4419. The OLS case
      4420. -
    4420. The Ridge case
      4421. -
    4421. Writing the Cost Function
      4422. -
    4422. Lasso case
      4423. -
    4423. The first Case
      4424. -
    4424. Simple code for solving the above problem
      4425. -
    4425. With Lasso Regression
      4426. -
    4426. Another Example, now with a polynomial fit
      4427. -
    4427. To think about, first part
      4428. -
    4428. More thinking
      4429. -
    4429. Still thinking
      4430. -
    4430. What does centering (subtracting the mean values) mean mathematically?
      4431. -
    4431. Further Manipulations
      4432. -
    4432. Wrapping it up
      4433. -
    4433. Linear Regression code, Intercept handling first
      4434. -
    4434. Code Examples
      4435. -
    4435. Taking out the mean
      4436. -
    4436. Friday September 9
      4437. -
    4437. Linking the regression analysis with a statistical interpretation
      4438. -
    4438. Assumptions made
      4439. -
    4439. Expectation value and variance
      4440. -
    4440. Expectation value and variance for \( \boldsymbol{\beta} \)
      4441. -
    4441. Deriving OLS from a probability distribution
      4442. -
    4442. Independent and Identically Distrubuted (iid)
      4443. -
    4443. Maximum Likelihood Estimation (MLE)
      4444. -
    4444. A new Cost Function
      4445. -
    4445. More basic Statistics and Bayes' theorem
      4446. -
    4446. Marginal Probability
      4447. -
    4447. Conditional Probability
      4448. -
    4448. Bayes' Theorem
      4449. -
    4449. Interpretations of Bayes' Theorem
      4450. -
    4450. Example of Usage of Bayes' theorem
      4451. -
    4451. Doing it correctly
      4452. -
    4452. Bayes' Theorem and Ridge and Lasso Regression
      4453. -
    4453. Test Function for what happens with OLS, Ridge and Lasso
      4454. -
    4454. Invoking Bayes' theorem
      4455. -
    4455. Ridge and Bayes
      4456. -
    4456. Lasso and Bayes
      4457. -
    4457. Exercise 1: mean values and variances in linear regression
      4458. -
    4458. Exercise 2: Adding Ridge and Lasso Regression
      4459. +
    4459. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4460. +
    4460. Linear Regression and the SVD
      4461. +
    4461. What does it mean?
      4462. +
    4462. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4463. +
    4463. Code for SVD and Inversion of Matrices
      4464. +
    4464. Inverse of Rectangular Matrix
      4465. +
    4465. Ridge and LASSO Regression
      4466. +
    4466. From OLS to Ridge and Lasso
      4467. +
    4467. Deriving the Ridge Regression Equations
      4468. +
    4468. Note on Scikit-Learn
      4469. +
    4469. Comparison with OLS
      4470. +
    4470. SVD analysis
      4471. +
    4471. Interpreting the Ridge results
      4472. +
    4472. More interpretations
      4473. +
    4473. Deriving the Lasso Regression Equations
      4474. +
    4474. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4475. +
    4475. Ridge Regression
      4476. +
    4476. Lasso Regression
      4477. +
    4477. Yet another Example
      4478. +
    4478. The OLS case
      4479. +
    4479. The Ridge case
      4480. +
    4480. Writing the Cost Function
      4481. +
    4481. Lasso case
      4482. +
    4482. The first Case
      4483. +
    4483. Simple code for solving the above problem
      4484. +
    4484. With Lasso Regression
      4485. +
    4485. Another Example, now with a polynomial fit
      4486. +
    4486. To think about, first part
      4487. +
    4487. More thinking
      4488. +
    4488. Still thinking
      4489. +
    4489. What does centering (subtracting the mean values) mean mathematically?
      4490. +
    4490. Further Manipulations
      4491. +
    4491. Wrapping it up
      4492. +
    4492. Linear Regression code, Intercept handling first
      4493. +
    4493. Code Examples
      4494. +
    4494. Taking out the mean
      4495. +
    4495. Friday September 9
      4496. +
    4496. Linking the regression analysis with a statistical interpretation
      4497. +
    4497. Assumptions made
      4498. +
    4498. Expectation value and variance
      4499. +
    4499. Expectation value and variance for \( \boldsymbol{\beta} \)
      4500. +
    4500. Deriving OLS from a probability distribution
      4501. +
    4501. Independent and Identically Distrubuted (iid)
      4502. +
    4502. Maximum Likelihood Estimation (MLE)
      4503. +
    4503. A new Cost Function
      4504. +
    4504. More basic Statistics and Bayes' theorem
      4505. +
    4505. Marginal Probability
      4506. +
    4506. Conditional Probability
      4507. +
    4507. Bayes' Theorem
      4508. +
    4508. Interpretations of Bayes' Theorem
      4509. +
    4509. Example of Usage of Bayes' theorem
      4510. +
    4510. Doing it correctly
      4511. +
    4511. Bayes' Theorem and Ridge and Lasso Regression
      4512. +
    4512. Test Function for what happens with OLS, Ridge and Lasso
      4513. +
    4513. Invoking Bayes' theorem
      4514. +
    4514. Ridge and Bayes
      4515. +
    4515. Lasso and Bayes
      4516. +
    4516. Exercise 1: mean values and variances in linear regression
      4517. +
    4517. Exercise 2: Adding Ridge and Lasso Regression
      4518. @@ -297,10 +295,7 @@ MathJax.Hub.Config({

       

       

       

      -

      Code Examples

      - -

      Armed with this wisdom, we attempt first to simply set the intercept equal to False in our implementation of Ridge regression for our well-known vanilla data set.

      - +

      Taking out the mean

      @@ -313,62 +308,75 @@ MathJax.Hub.Config({ import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn import linear_model +from sklearn.preprocessing import StandardScaler def MSE(y_data,y_model): n = np.size(y_model) return np.sum((y_data-y_model)**2)/n - - # A seed just to ensure that the random numbers are the same for every run. # Useful for eventual debugging. -np.random.seed(3155) +np.random.seed(315) n = 100 x = np.random.rand(n) y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) Maxpolydegree = 20 -X = np.zeros((n,Maxpolydegree)) -#We include explicitely the intercept column -for degree in range(Maxpolydegree): - X[:,degree] = x**degree +X = np.zeros((n,Maxpolydegree-1)) + +for degree in range(1,Maxpolydegree): #No intercept column + X[:,degree-1] = x**(degree) + # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -p = Maxpolydegree +#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable +X_train_mean = np.mean(X_train,axis=0) +#Center by removing mean from each feature +X_train_scaled = X_train - X_train_mean +X_test_scaled = X_test - X_train_mean +#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered) +#Remove the intercept from the training data. +y_scaler = np.mean(y_train) +y_train_scaled = y_train - y_scaler + +p = Maxpolydegree-1 I = np.eye(p,p) # Decide which values of lambda to use nlambdas = 6 MSEOwnRidgePredict = np.zeros(nlambdas) MSERidgePredict = np.zeros(nlambdas) + lambdas = np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train - # Note: we include the intercept column and no scaling - RegRidge = linear_model.Ridge(lmb,fit_intercept=False) + OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) + intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data + #Add intercept to prediction + ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler + RegRidge = linear_model.Ridge(lmb) RegRidge.fit(X_train,y_train) - # and then make the prediction - ytildeOwnRidge = X_train @ OwnRidgeBeta - ypredictOwnRidge = X_test @ OwnRidgeBeta - ytildeRidge = RegRidge.predict(X_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) + print(OwnRidgeBeta) #Intercept is given by mean of target variable print("Beta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) + print('Intercept from own implementation:') + print(intercept_) + print('Intercept from Scikit-Learn Ridge implementation') + print(RegRidge.intercept_) print("MSE values for own Ridge implementation") print(MSEOwnRidgePredict[i]) print("MSE values for Scikit-Learn Ridge implementation") print(MSERidgePredict[i]) + # Now plot the results plt.figure() -plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test') -plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test') - +plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'b--', label = 'MSE own Ridge Test') +plt.plot(np.log10(lambdas), MSERidgePredict, 'g--', label = 'MSE SL Ridge Test') plt.xlabel('log10(lambda)') plt.ylabel('MSE') plt.legend() @@ -388,10 +396,14 @@ plt.show()
      -

      The results here agree when we force Scikit-Learn's Ridge function to include the first column in our design matrix. -We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix. -What happens if we do not include the intercept in our fit? -Let us see how we can change this code by zero centering (thanks to Stian Bilek for inpouts here). +

      We see here, when compared to the code which includes explicitely the +intercept column, that our MSE value is actually smaller. This is +because the regularization term does not include the intercept value +\( \beta_0 \) in the fitting. This applies to Lasso regularization as +well. It means that our optimization is now done only with the +centered matrix and/or vector that enter the fitting procedure. Note +also that the problem with the intercept occurs mainly in these type +of polynomial fitting problem.

      @@ -419,7 +431,7 @@ Let us see how we can change this code by zero centering (thanks to Stian Bilek

    4518. 46
    4519. 47
    4520. ...
      4521. -
    4521. 60
      4522. +
    4522. 59
    4523. »
      4524. diff --git a/doc/pub/week36/html/._week36-bs038.html b/doc/pub/week36/html/._week36-bs038.html index 6d2422437..767fb68fd 100644 --- a/doc/pub/week36/html/._week36-bs038.html +++ b/doc/pub/week36/html/._week36-bs038.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    4524. Plans for week 36
      4525. -
    4525. Thursday September 8
      4526. -
    4526. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4527. -
    4527. Linear Regression and the SVD
      4528. -
    4528. What does it mean?
      4529. -
    4529. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4530. -
    4530. Code for SVD and Inversion of Matrices
      4531. -
    4531. Inverse of Rectangular Matrix
      4532. -
    4532. Ridge and LASSO Regression
      4533. -
    4533. From OLS to Ridge and Lasso
      4534. -
    4534. Deriving the Ridge Regression Equations
      4535. -
    4535. Note on Scikit-Learn
      4536. -
    4536. Comparison with OLS
      4537. -
    4537. SVD analysis
      4538. -
    4538. Interpreting the Ridge results
      4539. -
    4539. More interpretations
      4540. -
    4540. Deriving the Lasso Regression Equations
      4541. -
    4541. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4542. -
    4542. Ridge Regression
      4543. -
    4543. Lasso Regression
      4544. -
    4544. Yet another Example
      4545. -
    4545. The OLS case
      4546. -
    4546. The Ridge case
      4547. -
    4547. Writing the Cost Function
      4548. -
    4548. Lasso case
      4549. -
    4549. The first Case
      4550. -
    4550. Simple code for solving the above problem
      4551. -
    4551. With Lasso Regression
      4552. -
    4552. Another Example, now with a polynomial fit
      4553. -
    4553. To think about, first part
      4554. -
    4554. More thinking
      4555. -
    4555. Still thinking
      4556. -
    4556. What does centering (subtracting the mean values) mean mathematically?
      4557. -
    4557. Further Manipulations
      4558. -
    4558. Wrapping it up
      4559. -
    4559. Linear Regression code, Intercept handling first
      4560. -
    4560. Code Examples
      4561. -
    4561. Taking out the mean
      4562. -
    4562. Friday September 9
      4563. -
    4563. Linking the regression analysis with a statistical interpretation
      4564. -
    4564. Assumptions made
      4565. -
    4565. Expectation value and variance
      4566. -
    4566. Expectation value and variance for \( \boldsymbol{\beta} \)
      4567. -
    4567. Deriving OLS from a probability distribution
      4568. -
    4568. Independent and Identically Distrubuted (iid)
      4569. -
    4569. Maximum Likelihood Estimation (MLE)
      4570. -
    4570. A new Cost Function
      4571. -
    4571. More basic Statistics and Bayes' theorem
      4572. -
    4572. Marginal Probability
      4573. -
    4573. Conditional Probability
      4574. -
    4574. Bayes' Theorem
      4575. -
    4575. Interpretations of Bayes' Theorem
      4576. -
    4576. Example of Usage of Bayes' theorem
      4577. -
    4577. Doing it correctly
      4578. -
    4578. Bayes' Theorem and Ridge and Lasso Regression
      4579. -
    4579. Test Function for what happens with OLS, Ridge and Lasso
      4580. -
    4580. Invoking Bayes' theorem
      4581. -
    4581. Ridge and Bayes
      4582. -
    4582. Lasso and Bayes
      4583. -
    4583. Exercise 1: mean values and variances in linear regression
      4584. -
    4584. Exercise 2: Adding Ridge and Lasso Regression
      4585. +
    4585. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4586. +
    4586. Linear Regression and the SVD
      4587. +
    4587. What does it mean?
      4588. +
    4588. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4589. +
    4589. Code for SVD and Inversion of Matrices
      4590. +
    4590. Inverse of Rectangular Matrix
      4591. +
    4591. Ridge and LASSO Regression
      4592. +
    4592. From OLS to Ridge and Lasso
      4593. +
    4593. Deriving the Ridge Regression Equations
      4594. +
    4594. Note on Scikit-Learn
      4595. +
    4595. Comparison with OLS
      4596. +
    4596. SVD analysis
      4597. +
    4597. Interpreting the Ridge results
      4598. +
    4598. More interpretations
      4599. +
    4599. Deriving the Lasso Regression Equations
      4600. +
    4600. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4601. +
    4601. Ridge Regression
      4602. +
    4602. Lasso Regression
      4603. +
    4603. Yet another Example
      4604. +
    4604. The OLS case
      4605. +
    4605. The Ridge case
      4606. +
    4606. Writing the Cost Function
      4607. +
    4607. Lasso case
      4608. +
    4608. The first Case
      4609. +
    4609. Simple code for solving the above problem
      4610. +
    4610. With Lasso Regression
      4611. +
    4611. Another Example, now with a polynomial fit
      4612. +
    4612. To think about, first part
      4613. +
    4613. More thinking
      4614. +
    4614. Still thinking
      4615. +
    4615. What does centering (subtracting the mean values) mean mathematically?
      4616. +
    4616. Further Manipulations
      4617. +
    4617. Wrapping it up
      4618. +
    4618. Linear Regression code, Intercept handling first
      4619. +
    4619. Code Examples
      4620. +
    4620. Taking out the mean
      4621. +
    4621. Friday September 9
      4622. +
    4622. Linking the regression analysis with a statistical interpretation
      4623. +
    4623. Assumptions made
      4624. +
    4624. Expectation value and variance
      4625. +
    4625. Expectation value and variance for \( \boldsymbol{\beta} \)
      4626. +
    4626. Deriving OLS from a probability distribution
      4627. +
    4627. Independent and Identically Distrubuted (iid)
      4628. +
    4628. Maximum Likelihood Estimation (MLE)
      4629. +
    4629. A new Cost Function
      4630. +
    4630. More basic Statistics and Bayes' theorem
      4631. +
    4631. Marginal Probability
      4632. +
    4632. Conditional Probability
      4633. +
    4633. Bayes' Theorem
      4634. +
    4634. Interpretations of Bayes' Theorem
      4635. +
    4635. Example of Usage of Bayes' theorem
      4636. +
    4636. Doing it correctly
      4637. +
    4637. Bayes' Theorem and Ridge and Lasso Regression
      4638. +
    4638. Test Function for what happens with OLS, Ridge and Lasso
      4639. +
    4639. Invoking Bayes' theorem
      4640. +
    4640. Ridge and Bayes
      4641. +
    4641. Lasso and Bayes
      4642. +
    4642. Exercise 1: mean values and variances in linear regression
      4643. +
    4643. Exercise 2: Adding Ridge and Lasso Regression
      4644. @@ -297,116 +295,7 @@ MathJax.Hub.Config({

       

       

       

      -

      Taking out the mean

      - - -
      -
      -
      -
      -
      - 
      import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-from sklearn import linear_model
-from sklearn.preprocessing import StandardScaler
-
-def MSE(y_data,y_model):
-    n = np.size(y_model)
-    return np.sum((y_data-y_model)**2)/n
-# A seed just to ensure that the random numbers are the same for every run.
-# Useful for eventual debugging.
-np.random.seed(315)
-
-n = 100
-x = np.random.rand(n)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)
-
-Maxpolydegree = 20
-X = np.zeros((n,Maxpolydegree-1))
-
-for degree in range(1,Maxpolydegree): #No intercept column
-    X[:,degree-1] = x**(degree)
-
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-
-#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable
-X_train_mean = np.mean(X_train,axis=0)
-#Center by removing mean from each feature
-X_train_scaled = X_train - X_train_mean 
-X_test_scaled = X_test - X_train_mean
-#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered)
-#Remove the intercept from the training data.
-y_scaler = np.mean(y_train)           
-y_train_scaled = y_train - y_scaler   
-
-p = Maxpolydegree-1
-I = np.eye(p,p)
-# Decide which values of lambda to use
-nlambdas = 6
-MSEOwnRidgePredict = np.zeros(nlambdas)
-MSERidgePredict = np.zeros(nlambdas)
-
-lambdas = np.logspace(-4, 2, nlambdas)
-for i in range(nlambdas):
-    lmb = lambdas[i]
-    OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)
-    intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data
-    #Add intercept to prediction
-    ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler 
-    RegRidge = linear_model.Ridge(lmb)
-    RegRidge.fit(X_train,y_train)
-    ypredictRidge = RegRidge.predict(X_test)
-    MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
-    MSERidgePredict[i] = MSE(y_test,ypredictRidge)
-    print("Beta values for own Ridge implementation")
-    print(OwnRidgeBeta) #Intercept is given by mean of target variable
-    print("Beta values for Scikit-Learn Ridge implementation")
-    print(RegRidge.coef_)
-    print('Intercept from own implementation:')
-    print(intercept_)
-    print('Intercept from Scikit-Learn Ridge implementation')
-    print(RegRidge.intercept_)
-    print("MSE values for own Ridge implementation")
-    print(MSEOwnRidgePredict[i])
-    print("MSE values for Scikit-Learn Ridge implementation")
-    print(MSERidgePredict[i])
-
-
-# Now plot the results
-plt.figure()
-plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'b--', label = 'MSE own Ridge Test')
-plt.plot(np.log10(lambdas), MSERidgePredict, 'g--', label = 'MSE SL Ridge Test')
-plt.xlabel('log10(lambda)')
-plt.ylabel('MSE')
-plt.legend()
-plt.show()
-
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      - -

      We see here, when compared to the code which includes explicitely the -intercept column, that our MSE value is actually smaller. This is -because the regularization term does not include the intercept value -\( \beta_0 \) in the fitting. This applies to Lasso regularization as -well. It means that our optimization is now done only with the -centered matrix and/or vector that enter the fitting procedure. Note -also that the problem with the intercept occurs mainly in these type -of polynomial fitting problem. -

      +

      Friday September 9

      @@ -433,7 +322,7 @@ of polynomial fitting problem.

    4644. 47
    4645. 48
    4646. ...
      4647. -
    4647. 60
      4648. +
    4648. 59
    4649. »
      4650. diff --git a/doc/pub/week36/html/._week36-bs039.html b/doc/pub/week36/html/._week36-bs039.html index bec8f70a3..f255d2e5a 100644 --- a/doc/pub/week36/html/._week36-bs039.html +++ b/doc/pub/week36/html/._week36-bs039.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    4650. Plans for week 36
      4651. -
    4651. Thursday September 8
      4652. -
    4652. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4653. -
    4653. Linear Regression and the SVD
      4654. -
    4654. What does it mean?
      4655. -
    4655. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4656. -
    4656. Code for SVD and Inversion of Matrices
      4657. -
    4657. Inverse of Rectangular Matrix
      4658. -
    4658. Ridge and LASSO Regression
      4659. -
    4659. From OLS to Ridge and Lasso
      4660. -
    4660. Deriving the Ridge Regression Equations
      4661. -
    4661. Note on Scikit-Learn
      4662. -
    4662. Comparison with OLS
      4663. -
    4663. SVD analysis
      4664. -
    4664. Interpreting the Ridge results
      4665. -
    4665. More interpretations
      4666. -
    4666. Deriving the Lasso Regression Equations
      4667. -
    4667. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4668. -
    4668. Ridge Regression
      4669. -
    4669. Lasso Regression
      4670. -
    4670. Yet another Example
      4671. -
    4671. The OLS case
      4672. -
    4672. The Ridge case
      4673. -
    4673. Writing the Cost Function
      4674. -
    4674. Lasso case
      4675. -
    4675. The first Case
      4676. -
    4676. Simple code for solving the above problem
      4677. -
    4677. With Lasso Regression
      4678. -
    4678. Another Example, now with a polynomial fit
      4679. -
    4679. To think about, first part
      4680. -
    4680. More thinking
      4681. -
    4681. Still thinking
      4682. -
    4682. What does centering (subtracting the mean values) mean mathematically?
      4683. -
    4683. Further Manipulations
      4684. -
    4684. Wrapping it up
      4685. -
    4685. Linear Regression code, Intercept handling first
      4686. -
    4686. Code Examples
      4687. -
    4687. Taking out the mean
      4688. -
    4688. Friday September 9
      4689. -
    4689. Linking the regression analysis with a statistical interpretation
      4690. -
    4690. Assumptions made
      4691. -
    4691. Expectation value and variance
      4692. -
    4692. Expectation value and variance for \( \boldsymbol{\beta} \)
      4693. -
    4693. Deriving OLS from a probability distribution
      4694. -
    4694. Independent and Identically Distrubuted (iid)
      4695. -
    4695. Maximum Likelihood Estimation (MLE)
      4696. -
    4696. A new Cost Function
      4697. -
    4697. More basic Statistics and Bayes' theorem
      4698. -
    4698. Marginal Probability
      4699. -
    4699. Conditional Probability
      4700. -
    4700. Bayes' Theorem
      4701. -
    4701. Interpretations of Bayes' Theorem
      4702. -
    4702. Example of Usage of Bayes' theorem
      4703. -
    4703. Doing it correctly
      4704. -
    4704. Bayes' Theorem and Ridge and Lasso Regression
      4705. -
    4705. Test Function for what happens with OLS, Ridge and Lasso
      4706. -
    4706. Invoking Bayes' theorem
      4707. -
    4707. Ridge and Bayes
      4708. -
    4708. Lasso and Bayes
      4709. -
    4709. Exercise 1: mean values and variances in linear regression
      4710. -
    4710. Exercise 2: Adding Ridge and Lasso Regression
      4711. +
    4711. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4712. +
    4712. Linear Regression and the SVD
      4713. +
    4713. What does it mean?
      4714. +
    4714. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4715. +
    4715. Code for SVD and Inversion of Matrices
      4716. +
    4716. Inverse of Rectangular Matrix
      4717. +
    4717. Ridge and LASSO Regression
      4718. +
    4718. From OLS to Ridge and Lasso
      4719. +
    4719. Deriving the Ridge Regression Equations
      4720. +
    4720. Note on Scikit-Learn
      4721. +
    4721. Comparison with OLS
      4722. +
    4722. SVD analysis
      4723. +
    4723. Interpreting the Ridge results
      4724. +
    4724. More interpretations
      4725. +
    4725. Deriving the Lasso Regression Equations
      4726. +
    4726. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4727. +
    4727. Ridge Regression
      4728. +
    4728. Lasso Regression
      4729. +
    4729. Yet another Example
      4730. +
    4730. The OLS case
      4731. +
    4731. The Ridge case
      4732. +
    4732. Writing the Cost Function
      4733. +
    4733. Lasso case
      4734. +
    4734. The first Case
      4735. +
    4735. Simple code for solving the above problem
      4736. +
    4736. With Lasso Regression
      4737. +
    4737. Another Example, now with a polynomial fit
      4738. +
    4738. To think about, first part
      4739. +
    4739. More thinking
      4740. +
    4740. Still thinking
      4741. +
    4741. What does centering (subtracting the mean values) mean mathematically?
      4742. +
    4742. Further Manipulations
      4743. +
    4743. Wrapping it up
      4744. +
    4744. Linear Regression code, Intercept handling first
      4745. +
    4745. Code Examples
      4746. +
    4746. Taking out the mean
      4747. +
    4747. Friday September 9
      4748. +
    4748. Linking the regression analysis with a statistical interpretation
      4749. +
    4749. Assumptions made
      4750. +
    4750. Expectation value and variance
      4751. +
    4751. Expectation value and variance for \( \boldsymbol{\beta} \)
      4752. +
    4752. Deriving OLS from a probability distribution
      4753. +
    4753. Independent and Identically Distrubuted (iid)
      4754. +
    4754. Maximum Likelihood Estimation (MLE)
      4755. +
    4755. A new Cost Function
      4756. +
    4756. More basic Statistics and Bayes' theorem
      4757. +
    4757. Marginal Probability
      4758. +
    4758. Conditional Probability
      4759. +
    4759. Bayes' Theorem
      4760. +
    4760. Interpretations of Bayes' Theorem
      4761. +
    4761. Example of Usage of Bayes' theorem
      4762. +
    4762. Doing it correctly
      4763. +
    4763. Bayes' Theorem and Ridge and Lasso Regression
      4764. +
    4764. Test Function for what happens with OLS, Ridge and Lasso
      4765. +
    4765. Invoking Bayes' theorem
      4766. +
    4766. Ridge and Bayes
      4767. +
    4767. Lasso and Bayes
      4768. +
    4768. Exercise 1: mean values and variances in linear regression
      4769. +
    4769. Exercise 2: Adding Ridge and Lasso Regression
      4770. @@ -296,8 +294,50 @@ MathJax.Hub.Config({

       

       

       

      - -

      Friday September 9

      + +

      Linking the regression analysis with a statistical interpretation

      + +

      We will now couple the discussions of ordinary least squares, Ridge +and Lasso regression with a statistical interpretation, that is we +move from a linear algebra analysis to a statistical analysis. In +particular, we will focus on what the regularization terms can result +in. We will amongst other things show that the regularization +parameter can reduce considerably the variance of the parameters +\( \beta \). +

      + +

      The +advantage of doing linear regression is that we actually end up with +analytical expressions for several statistical quantities. +Standard least squares and Ridge regression allow us to +derive quantities like the variance and other expectation values in a +rather straightforward way. +

      + +

      It is assumed that \( \varepsilon_i +\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are +independent, i.e.: +

      +$$ +\begin{align*} +\mbox{Cov}(\varepsilon_{i_1}, +\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} +& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. +\end{align*} +$$ + +

      The randomness of \( \varepsilon_i \) implies that +\( \mathbf{y}_i \) is also a random variable. In particular, +\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim +\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a +non-random scalar. To specify the parameters of the distribution of +\( \mathbf{y}_i \) we need to calculate its first two moments. +

      + +

      Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The +notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the +row number \( i \) and perform a sum over all values \( p \). +

      @@ -324,7 +364,7 @@ MathJax.Hub.Config({

    4770. 48
    4771. 49
    4772. ...
      4773. -
    4773. 60
      4774. +
    4774. 59
    4775. »
      4776. diff --git a/doc/pub/week36/html/._week36-bs040.html b/doc/pub/week36/html/._week36-bs040.html index 338bd46f3..d09b6f9cd 100644 --- a/doc/pub/week36/html/._week36-bs040.html +++ b/doc/pub/week36/html/._week36-bs040.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    4776. Plans for week 36
      4777. -
    4777. Thursday September 8
      4778. -
    4778. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4779. -
    4779. Linear Regression and the SVD
      4780. -
    4780. What does it mean?
      4781. -
    4781. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4782. -
    4782. Code for SVD and Inversion of Matrices
      4783. -
    4783. Inverse of Rectangular Matrix
      4784. -
    4784. Ridge and LASSO Regression
      4785. -
    4785. From OLS to Ridge and Lasso
      4786. -
    4786. Deriving the Ridge Regression Equations
      4787. -
    4787. Note on Scikit-Learn
      4788. -
    4788. Comparison with OLS
      4789. -
    4789. SVD analysis
      4790. -
    4790. Interpreting the Ridge results
      4791. -
    4791. More interpretations
      4792. -
    4792. Deriving the Lasso Regression Equations
      4793. -
    4793. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4794. -
    4794. Ridge Regression
      4795. -
    4795. Lasso Regression
      4796. -
    4796. Yet another Example
      4797. -
    4797. The OLS case
      4798. -
    4798. The Ridge case
      4799. -
    4799. Writing the Cost Function
      4800. -
    4800. Lasso case
      4801. -
    4801. The first Case
      4802. -
    4802. Simple code for solving the above problem
      4803. -
    4803. With Lasso Regression
      4804. -
    4804. Another Example, now with a polynomial fit
      4805. -
    4805. To think about, first part
      4806. -
    4806. More thinking
      4807. -
    4807. Still thinking
      4808. -
    4808. What does centering (subtracting the mean values) mean mathematically?
      4809. -
    4809. Further Manipulations
      4810. -
    4810. Wrapping it up
      4811. -
    4811. Linear Regression code, Intercept handling first
      4812. -
    4812. Code Examples
      4813. -
    4813. Taking out the mean
      4814. -
    4814. Friday September 9
      4815. -
    4815. Linking the regression analysis with a statistical interpretation
      4816. -
    4816. Assumptions made
      4817. -
    4817. Expectation value and variance
      4818. -
    4818. Expectation value and variance for \( \boldsymbol{\beta} \)
      4819. -
    4819. Deriving OLS from a probability distribution
      4820. -
    4820. Independent and Identically Distrubuted (iid)
      4821. -
    4821. Maximum Likelihood Estimation (MLE)
      4822. -
    4822. A new Cost Function
      4823. -
    4823. More basic Statistics and Bayes' theorem
      4824. -
    4824. Marginal Probability
      4825. -
    4825. Conditional Probability
      4826. -
    4826. Bayes' Theorem
      4827. -
    4827. Interpretations of Bayes' Theorem
      4828. -
    4828. Example of Usage of Bayes' theorem
      4829. -
    4829. Doing it correctly
      4830. -
    4830. Bayes' Theorem and Ridge and Lasso Regression
      4831. -
    4831. Test Function for what happens with OLS, Ridge and Lasso
      4832. -
    4832. Invoking Bayes' theorem
      4833. -
    4833. Ridge and Bayes
      4834. -
    4834. Lasso and Bayes
      4835. -
    4835. Exercise 1: mean values and variances in linear regression
      4836. -
    4836. Exercise 2: Adding Ridge and Lasso Regression
      4837. +
    4837. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4838. +
    4838. Linear Regression and the SVD
      4839. +
    4839. What does it mean?
      4840. +
    4840. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4841. +
    4841. Code for SVD and Inversion of Matrices
      4842. +
    4842. Inverse of Rectangular Matrix
      4843. +
    4843. Ridge and LASSO Regression
      4844. +
    4844. From OLS to Ridge and Lasso
      4845. +
    4845. Deriving the Ridge Regression Equations
      4846. +
    4846. Note on Scikit-Learn
      4847. +
    4847. Comparison with OLS
      4848. +
    4848. SVD analysis
      4849. +
    4849. Interpreting the Ridge results
      4850. +
    4850. More interpretations
      4851. +
    4851. Deriving the Lasso Regression Equations
      4852. +
    4852. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4853. +
    4853. Ridge Regression
      4854. +
    4854. Lasso Regression
      4855. +
    4855. Yet another Example
      4856. +
    4856. The OLS case
      4857. +
    4857. The Ridge case
      4858. +
    4858. Writing the Cost Function
      4859. +
    4859. Lasso case
      4860. +
    4860. The first Case
      4861. +
    4861. Simple code for solving the above problem
      4862. +
    4862. With Lasso Regression
      4863. +
    4863. Another Example, now with a polynomial fit
      4864. +
    4864. To think about, first part
      4865. +
    4865. More thinking
      4866. +
    4866. Still thinking
      4867. +
    4867. What does centering (subtracting the mean values) mean mathematically?
      4868. +
    4868. Further Manipulations
      4869. +
    4869. Wrapping it up
      4870. +
    4870. Linear Regression code, Intercept handling first
      4871. +
    4871. Code Examples
      4872. +
    4872. Taking out the mean
      4873. +
    4873. Friday September 9
      4874. +
    4874. Linking the regression analysis with a statistical interpretation
      4875. +
    4875. Assumptions made
      4876. +
    4876. Expectation value and variance
      4877. +
    4877. Expectation value and variance for \( \boldsymbol{\beta} \)
      4878. +
    4878. Deriving OLS from a probability distribution
      4879. +
    4879. Independent and Identically Distrubuted (iid)
      4880. +
    4880. Maximum Likelihood Estimation (MLE)
      4881. +
    4881. A new Cost Function
      4882. +
    4882. More basic Statistics and Bayes' theorem
      4883. +
    4883. Marginal Probability
      4884. +
    4884. Conditional Probability
      4885. +
    4885. Bayes' Theorem
      4886. +
    4886. Interpretations of Bayes' Theorem
      4887. +
    4887. Example of Usage of Bayes' theorem
      4888. +
    4888. Doing it correctly
      4889. +
    4889. Bayes' Theorem and Ridge and Lasso Regression
      4890. +
    4890. Test Function for what happens with OLS, Ridge and Lasso
      4891. +
    4891. Invoking Bayes' theorem
      4892. +
    4892. Ridge and Bayes
      4893. +
    4893. Lasso and Bayes
      4894. +
    4894. Exercise 1: mean values and variances in linear regression
      4895. +
    4895. Exercise 2: Adding Ridge and Lasso Regression
      4896. @@ -296,50 +294,24 @@ MathJax.Hub.Config({

       

       

       

      - -

      Linking the regression analysis with a statistical interpretation

      + +

      Assumptions made

      -

      We will now couple the discussions of ordinary least squares, Ridge -and Lasso regression with a statistical interpretation, that is we -move from a linear algebra analysis to a statistical analysis. In -particular, we will focus on what the regularization terms can result -in. We will amongst other things show that the regularization -parameter can reduce considerably the variance of the parameters -\( \beta \). -

      - -

      The -advantage of doing linear regression is that we actually end up with -analytical expressions for several statistical quantities. -Standard least squares and Ridge regression allow us to -derive quantities like the variance and other expectation values in a -rather straightforward way. -

      - -

      It is assumed that \( \varepsilon_i -\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are -independent, i.e.: +

      The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off) +that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) +which describe our data

      $$ -\begin{align*} -\mbox{Cov}(\varepsilon_{i_1}, -\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} -& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. -\end{align*} +\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} $$ -

      The randomness of \( \varepsilon_i \) implies that -\( \mathbf{y}_i \) is also a random variable. In particular, -\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim -\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a -non-random scalar. To specify the parameters of the distribution of -\( \mathbf{y}_i \) we need to calculate its first two moments. +

      We approximate this function with our model from the solution of the linear regression equations, that is our +function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with

      +$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. +$$ -

      Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The -notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the -row number \( i \) and perform a sum over all values \( p \). -

      @@ -366,7 +338,7 @@ row number \( i \) and perform a sum over all values \( p \).

    4896. 49
    4897. 50
    4898. ...
      4899. -
    4899. 60
      4900. +
    4900. 59
    4901. »
      4902. diff --git a/doc/pub/week36/html/._week36-bs041.html b/doc/pub/week36/html/._week36-bs041.html index 711c5de52..3a6cf832a 100644 --- a/doc/pub/week36/html/._week36-bs041.html +++ b/doc/pub/week36/html/._week36-bs041.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    4902. Plans for week 36
      4903. -
    4903. Thursday September 8
      4904. -
    4904. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4905. -
    4905. Linear Regression and the SVD
      4906. -
    4906. What does it mean?
      4907. -
    4907. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4908. -
    4908. Code for SVD and Inversion of Matrices
      4909. -
    4909. Inverse of Rectangular Matrix
      4910. -
    4910. Ridge and LASSO Regression
      4911. -
    4911. From OLS to Ridge and Lasso
      4912. -
    4912. Deriving the Ridge Regression Equations
      4913. -
    4913. Note on Scikit-Learn
      4914. -
    4914. Comparison with OLS
      4915. -
    4915. SVD analysis
      4916. -
    4916. Interpreting the Ridge results
      4917. -
    4917. More interpretations
      4918. -
    4918. Deriving the Lasso Regression Equations
      4919. -
    4919. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4920. -
    4920. Ridge Regression
      4921. -
    4921. Lasso Regression
      4922. -
    4922. Yet another Example
      4923. -
    4923. The OLS case
      4924. -
    4924. The Ridge case
      4925. -
    4925. Writing the Cost Function
      4926. -
    4926. Lasso case
      4927. -
    4927. The first Case
      4928. -
    4928. Simple code for solving the above problem
      4929. -
    4929. With Lasso Regression
      4930. -
    4930. Another Example, now with a polynomial fit
      4931. -
    4931. To think about, first part
      4932. -
    4932. More thinking
      4933. -
    4933. Still thinking
      4934. -
    4934. What does centering (subtracting the mean values) mean mathematically?
      4935. -
    4935. Further Manipulations
      4936. -
    4936. Wrapping it up
      4937. -
    4937. Linear Regression code, Intercept handling first
      4938. -
    4938. Code Examples
      4939. -
    4939. Taking out the mean
      4940. -
    4940. Friday September 9
      4941. -
    4941. Linking the regression analysis with a statistical interpretation
      4942. -
    4942. Assumptions made
      4943. -
    4943. Expectation value and variance
      4944. -
    4944. Expectation value and variance for \( \boldsymbol{\beta} \)
      4945. -
    4945. Deriving OLS from a probability distribution
      4946. -
    4946. Independent and Identically Distrubuted (iid)
      4947. -
    4947. Maximum Likelihood Estimation (MLE)
      4948. -
    4948. A new Cost Function
      4949. -
    4949. More basic Statistics and Bayes' theorem
      4950. -
    4950. Marginal Probability
      4951. -
    4951. Conditional Probability
      4952. -
    4952. Bayes' Theorem
      4953. -
    4953. Interpretations of Bayes' Theorem
      4954. -
    4954. Example of Usage of Bayes' theorem
      4955. -
    4955. Doing it correctly
      4956. -
    4956. Bayes' Theorem and Ridge and Lasso Regression
      4957. -
    4957. Test Function for what happens with OLS, Ridge and Lasso
      4958. -
    4958. Invoking Bayes' theorem
      4959. -
    4959. Ridge and Bayes
      4960. -
    4960. Lasso and Bayes
      4961. -
    4961. Exercise 1: mean values and variances in linear regression
      4962. -
    4962. Exercise 2: Adding Ridge and Lasso Regression
      4963. +
    4963. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      4964. +
    4964. Linear Regression and the SVD
      4965. +
    4965. What does it mean?
      4966. +
    4966. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      4967. +
    4967. Code for SVD and Inversion of Matrices
      4968. +
    4968. Inverse of Rectangular Matrix
      4969. +
    4969. Ridge and LASSO Regression
      4970. +
    4970. From OLS to Ridge and Lasso
      4971. +
    4971. Deriving the Ridge Regression Equations
      4972. +
    4972. Note on Scikit-Learn
      4973. +
    4973. Comparison with OLS
      4974. +
    4974. SVD analysis
      4975. +
    4975. Interpreting the Ridge results
      4976. +
    4976. More interpretations
      4977. +
    4977. Deriving the Lasso Regression Equations
      4978. +
    4978. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      4979. +
    4979. Ridge Regression
      4980. +
    4980. Lasso Regression
      4981. +
    4981. Yet another Example
      4982. +
    4982. The OLS case
      4983. +
    4983. The Ridge case
      4984. +
    4984. Writing the Cost Function
      4985. +
    4985. Lasso case
      4986. +
    4986. The first Case
      4987. +
    4987. Simple code for solving the above problem
      4988. +
    4988. With Lasso Regression
      4989. +
    4989. Another Example, now with a polynomial fit
      4990. +
    4990. To think about, first part
      4991. +
    4991. More thinking
      4992. +
    4992. Still thinking
      4993. +
    4993. What does centering (subtracting the mean values) mean mathematically?
      4994. +
    4994. Further Manipulations
      4995. +
    4995. Wrapping it up
      4996. +
    4996. Linear Regression code, Intercept handling first
      4997. +
    4997. Code Examples
      4998. +
    4998. Taking out the mean
      4999. +
    4999. Friday September 9
      5000. +
    5000. Linking the regression analysis with a statistical interpretation
      5001. +
    5001. Assumptions made
      5002. +
    5002. Expectation value and variance
      5003. +
    5003. Expectation value and variance for \( \boldsymbol{\beta} \)
      5004. +
    5004. Deriving OLS from a probability distribution
      5005. +
    5005. Independent and Identically Distrubuted (iid)
      5006. +
    5006. Maximum Likelihood Estimation (MLE)
      5007. +
    5007. A new Cost Function
      5008. +
    5008. More basic Statistics and Bayes' theorem
      5009. +
    5009. Marginal Probability
      5010. +
    5010. Conditional Probability
      5011. +
    5011. Bayes' Theorem
      5012. +
    5012. Interpretations of Bayes' Theorem
      5013. +
    5013. Example of Usage of Bayes' theorem
      5014. +
    5014. Doing it correctly
      5015. +
    5015. Bayes' Theorem and Ridge and Lasso Regression
      5016. +
    5016. Test Function for what happens with OLS, Ridge and Lasso
      5017. +
    5017. Invoking Bayes' theorem
      5018. +
    5018. Ridge and Bayes
      5019. +
    5019. Lasso and Bayes
      5020. +
    5020. Exercise 1: mean values and variances in linear regression
      5021. +
    5021. Exercise 2: Adding Ridge and Lasso Regression
      5022. @@ -297,23 +295,38 @@ MathJax.Hub.Config({

       

       

       

      -

      Assumptions made

      +

      Expectation value and variance

      -

      The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off) -that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) -which describe our data +

      We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)

      +$$ +\begin{align*} +\mathbb{E}(y_i) & = +\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\end{align*} +$$ + +

      while +its variance is

      $$ -\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} +\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i +- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - +[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, +\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 +\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, +\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. +\end{align*} $$ -

      We approximate this function with our model from the solution of the linear regression equations, that is our -function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with +

      Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).

      -$$ -\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. -$$ -

      @@ -340,7 +353,7 @@ $$

    5022. 50
    5023. 51
    5024. ...
      5025. -
    5025. 60
      5026. +
    5026. 59
    5027. »
      5028. diff --git a/doc/pub/week36/html/._week36-bs042.html b/doc/pub/week36/html/._week36-bs042.html index 446c5273b..05c78da57 100644 --- a/doc/pub/week36/html/._week36-bs042.html +++ b/doc/pub/week36/html/._week36-bs042.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    5028. Plans for week 36
      5029. -
    5029. Thursday September 8
      5030. -
    5030. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5031. -
    5031. Linear Regression and the SVD
      5032. -
    5032. What does it mean?
      5033. -
    5033. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5034. -
    5034. Code for SVD and Inversion of Matrices
      5035. -
    5035. Inverse of Rectangular Matrix
      5036. -
    5036. Ridge and LASSO Regression
      5037. -
    5037. From OLS to Ridge and Lasso
      5038. -
    5038. Deriving the Ridge Regression Equations
      5039. -
    5039. Note on Scikit-Learn
      5040. -
    5040. Comparison with OLS
      5041. -
    5041. SVD analysis
      5042. -
    5042. Interpreting the Ridge results
      5043. -
    5043. More interpretations
      5044. -
    5044. Deriving the Lasso Regression Equations
      5045. -
    5045. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5046. -
    5046. Ridge Regression
      5047. -
    5047. Lasso Regression
      5048. -
    5048. Yet another Example
      5049. -
    5049. The OLS case
      5050. -
    5050. The Ridge case
      5051. -
    5051. Writing the Cost Function
      5052. -
    5052. Lasso case
      5053. -
    5053. The first Case
      5054. -
    5054. Simple code for solving the above problem
      5055. -
    5055. With Lasso Regression
      5056. -
    5056. Another Example, now with a polynomial fit
      5057. -
    5057. To think about, first part
      5058. -
    5058. More thinking
      5059. -
    5059. Still thinking
      5060. -
    5060. What does centering (subtracting the mean values) mean mathematically?
      5061. -
    5061. Further Manipulations
      5062. -
    5062. Wrapping it up
      5063. -
    5063. Linear Regression code, Intercept handling first
      5064. -
    5064. Code Examples
      5065. -
    5065. Taking out the mean
      5066. -
    5066. Friday September 9
      5067. -
    5067. Linking the regression analysis with a statistical interpretation
      5068. -
    5068. Assumptions made
      5069. -
    5069. Expectation value and variance
      5070. -
    5070. Expectation value and variance for \( \boldsymbol{\beta} \)
      5071. -
    5071. Deriving OLS from a probability distribution
      5072. -
    5072. Independent and Identically Distrubuted (iid)
      5073. -
    5073. Maximum Likelihood Estimation (MLE)
      5074. -
    5074. A new Cost Function
      5075. -
    5075. More basic Statistics and Bayes' theorem
      5076. -
    5076. Marginal Probability
      5077. -
    5077. Conditional Probability
      5078. -
    5078. Bayes' Theorem
      5079. -
    5079. Interpretations of Bayes' Theorem
      5080. -
    5080. Example of Usage of Bayes' theorem
      5081. -
    5081. Doing it correctly
      5082. -
    5082. Bayes' Theorem and Ridge and Lasso Regression
      5083. -
    5083. Test Function for what happens with OLS, Ridge and Lasso
      5084. -
    5084. Invoking Bayes' theorem
      5085. -
    5085. Ridge and Bayes
      5086. -
    5086. Lasso and Bayes
      5087. -
    5087. Exercise 1: mean values and variances in linear regression
      5088. -
    5088. Exercise 2: Adding Ridge and Lasso Regression
      5089. +
    5089. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5090. +
    5090. Linear Regression and the SVD
      5091. +
    5091. What does it mean?
      5092. +
    5092. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5093. +
    5093. Code for SVD and Inversion of Matrices
      5094. +
    5094. Inverse of Rectangular Matrix
      5095. +
    5095. Ridge and LASSO Regression
      5096. +
    5096. From OLS to Ridge and Lasso
      5097. +
    5097. Deriving the Ridge Regression Equations
      5098. +
    5098. Note on Scikit-Learn
      5099. +
    5099. Comparison with OLS
      5100. +
    5100. SVD analysis
      5101. +
    5101. Interpreting the Ridge results
      5102. +
    5102. More interpretations
      5103. +
    5103. Deriving the Lasso Regression Equations
      5104. +
    5104. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5105. +
    5105. Ridge Regression
      5106. +
    5106. Lasso Regression
      5107. +
    5107. Yet another Example
      5108. +
    5108. The OLS case
      5109. +
    5109. The Ridge case
      5110. +
    5110. Writing the Cost Function
      5111. +
    5111. Lasso case
      5112. +
    5112. The first Case
      5113. +
    5113. Simple code for solving the above problem
      5114. +
    5114. With Lasso Regression
      5115. +
    5115. Another Example, now with a polynomial fit
      5116. +
    5116. To think about, first part
      5117. +
    5117. More thinking
      5118. +
    5118. Still thinking
      5119. +
    5119. What does centering (subtracting the mean values) mean mathematically?
      5120. +
    5120. Further Manipulations
      5121. +
    5121. Wrapping it up
      5122. +
    5122. Linear Regression code, Intercept handling first
      5123. +
    5123. Code Examples
      5124. +
    5124. Taking out the mean
      5125. +
    5125. Friday September 9
      5126. +
    5126. Linking the regression analysis with a statistical interpretation
      5127. +
    5127. Assumptions made
      5128. +
    5128. Expectation value and variance
      5129. +
    5129. Expectation value and variance for \( \boldsymbol{\beta} \)
      5130. +
    5130. Deriving OLS from a probability distribution
      5131. +
    5131. Independent and Identically Distrubuted (iid)
      5132. +
    5132. Maximum Likelihood Estimation (MLE)
      5133. +
    5133. A new Cost Function
      5134. +
    5134. More basic Statistics and Bayes' theorem
      5135. +
    5135. Marginal Probability
      5136. +
    5136. Conditional Probability
      5137. +
    5137. Bayes' Theorem
      5138. +
    5138. Interpretations of Bayes' Theorem
      5139. +
    5139. Example of Usage of Bayes' theorem
      5140. +
    5140. Doing it correctly
      5141. +
    5141. Bayes' Theorem and Ridge and Lasso Regression
      5142. +
    5142. Test Function for what happens with OLS, Ridge and Lasso
      5143. +
    5143. Invoking Bayes' theorem
      5144. +
    5144. Ridge and Bayes
      5145. +
    5145. Lasso and Bayes
      5146. +
    5146. Exercise 1: mean values and variances in linear regression
      5147. +
    5147. Exercise 2: Adding Ridge and Lasso Regression
      5148. @@ -297,37 +295,81 @@ MathJax.Hub.Config({

       

       

       

      -

      Expectation value and variance

      +

      Expectation value and variance for \( \boldsymbol{\beta} \)

      -

      We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)

      +

      With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\beta}} \) we can evaluate the expectation value

      $$ -\begin{align*} -\mathbb{E}(y_i) & = -\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) -\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, -\end{align*} +\mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. $$ -

      while -its variance is +

      This means that the estimator of the regression parameters is unbiased.

      + +

      We can also calculate the variance

      + +

      The variance of the optimal value \( \boldsymbol{\hat{\beta}} \) is

      +$$ +\begin{eqnarray*} +\mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} +\\ +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} +\\ +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% \\ +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T +\\ +& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, +\end{eqnarray*} +$$ + +

      where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 +\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the +variance of the estimate of the \( j \)-th regression coefficient: +\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to +construct a confidence interval for the estimates.

      + +

      In a similar way, we can obtain analytical expressions for say the +expectation values of the parameters \( \boldsymbol{\beta} \) and their variance +when we employ Ridge regression, allowing us again to define a confidence interval. +

      + +

      It is rather straightforward to show that

      $$ -\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i -- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - -[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, -\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & -= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i -\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, -\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 -\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + -\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 -\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, -\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. -\end{align*} +\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. $$ -

      Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with -mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). +

      We see clearly that +\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. +

      + +

      We can also compute the variance as

      + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +$$ + +

      and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero.

      + +

      With this, we can compute the difference

      + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +$$ + +

      The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.

      @@ -355,7 +397,7 @@ mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (n

    5148. 51
    5149. 52
    5150. ...
      5151. -
    5151. 60
      5152. +
    5152. 59
    5153. »
      5154. diff --git a/doc/pub/week36/html/._week36-bs043.html b/doc/pub/week36/html/._week36-bs043.html index 4e0a74c6d..54edcc8fb 100644 --- a/doc/pub/week36/html/._week36-bs043.html +++ b/doc/pub/week36/html/._week36-bs043.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    5154. Plans for week 36
      5155. -
    5155. Thursday September 8
      5156. -
    5156. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5157. -
    5157. Linear Regression and the SVD
      5158. -
    5158. What does it mean?
      5159. -
    5159. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5160. -
    5160. Code for SVD and Inversion of Matrices
      5161. -
    5161. Inverse of Rectangular Matrix
      5162. -
    5162. Ridge and LASSO Regression
      5163. -
    5163. From OLS to Ridge and Lasso
      5164. -
    5164. Deriving the Ridge Regression Equations
      5165. -
    5165. Note on Scikit-Learn
      5166. -
    5166. Comparison with OLS
      5167. -
    5167. SVD analysis
      5168. -
    5168. Interpreting the Ridge results
      5169. -
    5169. More interpretations
      5170. -
    5170. Deriving the Lasso Regression Equations
      5171. -
    5171. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5172. -
    5172. Ridge Regression
      5173. -
    5173. Lasso Regression
      5174. -
    5174. Yet another Example
      5175. -
    5175. The OLS case
      5176. -
    5176. The Ridge case
      5177. -
    5177. Writing the Cost Function
      5178. -
    5178. Lasso case
      5179. -
    5179. The first Case
      5180. -
    5180. Simple code for solving the above problem
      5181. -
    5181. With Lasso Regression
      5182. -
    5182. Another Example, now with a polynomial fit
      5183. -
    5183. To think about, first part
      5184. -
    5184. More thinking
      5185. -
    5185. Still thinking
      5186. -
    5186. What does centering (subtracting the mean values) mean mathematically?
      5187. -
    5187. Further Manipulations
      5188. -
    5188. Wrapping it up
      5189. -
    5189. Linear Regression code, Intercept handling first
      5190. -
    5190. Code Examples
      5191. -
    5191. Taking out the mean
      5192. -
    5192. Friday September 9
      5193. -
    5193. Linking the regression analysis with a statistical interpretation
      5194. -
    5194. Assumptions made
      5195. -
    5195. Expectation value and variance
      5196. -
    5196. Expectation value and variance for \( \boldsymbol{\beta} \)
      5197. -
    5197. Deriving OLS from a probability distribution
      5198. -
    5198. Independent and Identically Distrubuted (iid)
      5199. -
    5199. Maximum Likelihood Estimation (MLE)
      5200. -
    5200. A new Cost Function
      5201. -
    5201. More basic Statistics and Bayes' theorem
      5202. -
    5202. Marginal Probability
      5203. -
    5203. Conditional Probability
      5204. -
    5204. Bayes' Theorem
      5205. -
    5205. Interpretations of Bayes' Theorem
      5206. -
    5206. Example of Usage of Bayes' theorem
      5207. -
    5207. Doing it correctly
      5208. -
    5208. Bayes' Theorem and Ridge and Lasso Regression
      5209. -
    5209. Test Function for what happens with OLS, Ridge and Lasso
      5210. -
    5210. Invoking Bayes' theorem
      5211. -
    5211. Ridge and Bayes
      5212. -
    5212. Lasso and Bayes
      5213. -
    5213. Exercise 1: mean values and variances in linear regression
      5214. -
    5214. Exercise 2: Adding Ridge and Lasso Regression
      5215. +
    5215. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5216. +
    5216. Linear Regression and the SVD
      5217. +
    5217. What does it mean?
      5218. +
    5218. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5219. +
    5219. Code for SVD and Inversion of Matrices
      5220. +
    5220. Inverse of Rectangular Matrix
      5221. +
    5221. Ridge and LASSO Regression
      5222. +
    5222. From OLS to Ridge and Lasso
      5223. +
    5223. Deriving the Ridge Regression Equations
      5224. +
    5224. Note on Scikit-Learn
      5225. +
    5225. Comparison with OLS
      5226. +
    5226. SVD analysis
      5227. +
    5227. Interpreting the Ridge results
      5228. +
    5228. More interpretations
      5229. +
    5229. Deriving the Lasso Regression Equations
      5230. +
    5230. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5231. +
    5231. Ridge Regression
      5232. +
    5232. Lasso Regression
      5233. +
    5233. Yet another Example
      5234. +
    5234. The OLS case
      5235. +
    5235. The Ridge case
      5236. +
    5236. Writing the Cost Function
      5237. +
    5237. Lasso case
      5238. +
    5238. The first Case
      5239. +
    5239. Simple code for solving the above problem
      5240. +
    5240. With Lasso Regression
      5241. +
    5241. Another Example, now with a polynomial fit
      5242. +
    5242. To think about, first part
      5243. +
    5243. More thinking
      5244. +
    5244. Still thinking
      5245. +
    5245. What does centering (subtracting the mean values) mean mathematically?
      5246. +
    5246. Further Manipulations
      5247. +
    5247. Wrapping it up
      5248. +
    5248. Linear Regression code, Intercept handling first
      5249. +
    5249. Code Examples
      5250. +
    5250. Taking out the mean
      5251. +
    5251. Friday September 9
      5252. +
    5252. Linking the regression analysis with a statistical interpretation
      5253. +
    5253. Assumptions made
      5254. +
    5254. Expectation value and variance
      5255. +
    5255. Expectation value and variance for \( \boldsymbol{\beta} \)
      5256. +
    5256. Deriving OLS from a probability distribution
      5257. +
    5257. Independent and Identically Distrubuted (iid)
      5258. +
    5258. Maximum Likelihood Estimation (MLE)
      5259. +
    5259. A new Cost Function
      5260. +
    5260. More basic Statistics and Bayes' theorem
      5261. +
    5261. Marginal Probability
      5262. +
    5262. Conditional Probability
      5263. +
    5263. Bayes' Theorem
      5264. +
    5264. Interpretations of Bayes' Theorem
      5265. +
    5265. Example of Usage of Bayes' theorem
      5266. +
    5266. Doing it correctly
      5267. +
    5267. Bayes' Theorem and Ridge and Lasso Regression
      5268. +
    5268. Test Function for what happens with OLS, Ridge and Lasso
      5269. +
    5269. Invoking Bayes' theorem
      5270. +
    5270. Ridge and Bayes
      5271. +
    5271. Lasso and Bayes
      5272. +
    5272. Exercise 1: mean values and variances in linear regression
      5273. +
    5273. Exercise 2: Adding Ridge and Lasso Regression
      5274. @@ -297,82 +295,27 @@ MathJax.Hub.Config({

       

       

       

      -

      Expectation value and variance for \( \boldsymbol{\beta} \)

      +

      Deriving OLS from a probability distribution

      -

      With the OLS expressions for the optimal parameters \( \boldsymbol{\hat{\beta}} \) we can evaluate the expectation value

      -$$ -\mathbb{E}(\boldsymbol{\hat{\beta}}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. -$$ - -

      This means that the estimator of the regression parameters is unbiased.

      - -

      We can also calculate the variance

      - -

      The variance of the optimal value \( \boldsymbol{\hat{\beta}} \) is

      -$$ -\begin{eqnarray*} -\mbox{Var}(\boldsymbol{\hat{\beta}}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} -\\ -& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} -\\ -% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -% \\ -% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -% \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -\\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -% \\ -% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} -% \\ -% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T -\\ -& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, -\end{eqnarray*} -$$ - -

      where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = -\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + -\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 -\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the -variance of the estimate of the \( j \)-th regression coefficient: -\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to -construct a confidence interval for the estimates. +

      Our basic assumption when we derived the OLS equations was to assume +that our output is determined by a given continuous function +\( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal +distribution with zero mean value and an undetermined variance +\( \sigma^2 \).

      -

      In a similar way, we can obtain analytical expressions for say the -expectation values of the parameters \( \boldsymbol{\beta} \) and their variance -when we employ Ridge regression, allowing us again to define a confidence interval. +

      We found above that the outputs \( \boldsymbol{y} \) have a mean value given by +\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to +the design matrix are not stochastic variables, we can assume that the +probability distribution of our targets is also a normal distribution +but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a +single output \( y_i \) is given by the Gaussian distribution

      -

      It is rather straightforward to show that

      $$ -\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$ -

      We see clearly that -\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. -

      - -

      We can also compute the variance as

      - -$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, -$$ - -

      and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero.

      - -

      With this, we can compute the difference

      - -$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. -$$ - -

      The difference is non-negative definite since each component of the -matrix product is non-negative definite. -This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. -

      @@ -399,7 +342,7 @@ This means the variance we obtain with the standard OLS will always for \( \lamb

    5274. 52
    5275. 53
    5276. ...
      5277. -
    5277. 60
      5278. +
    5278. 59
    5279. »
      5280. diff --git a/doc/pub/week36/html/._week36-bs044.html b/doc/pub/week36/html/._week36-bs044.html index a006bb45f..5597ca0bc 100644 --- a/doc/pub/week36/html/._week36-bs044.html +++ b/doc/pub/week36/html/._week36-bs044.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    5280. Plans for week 36
      5281. -
    5281. Thursday September 8
      5282. -
    5282. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5283. -
    5283. Linear Regression and the SVD
      5284. -
    5284. What does it mean?
      5285. -
    5285. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5286. -
    5286. Code for SVD and Inversion of Matrices
      5287. -
    5287. Inverse of Rectangular Matrix
      5288. -
    5288. Ridge and LASSO Regression
      5289. -
    5289. From OLS to Ridge and Lasso
      5290. -
    5290. Deriving the Ridge Regression Equations
      5291. -
    5291. Note on Scikit-Learn
      5292. -
    5292. Comparison with OLS
      5293. -
    5293. SVD analysis
      5294. -
    5294. Interpreting the Ridge results
      5295. -
    5295. More interpretations
      5296. -
    5296. Deriving the Lasso Regression Equations
      5297. -
    5297. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5298. -
    5298. Ridge Regression
      5299. -
    5299. Lasso Regression
      5300. -
    5300. Yet another Example
      5301. -
    5301. The OLS case
      5302. -
    5302. The Ridge case
      5303. -
    5303. Writing the Cost Function
      5304. -
    5304. Lasso case
      5305. -
    5305. The first Case
      5306. -
    5306. Simple code for solving the above problem
      5307. -
    5307. With Lasso Regression
      5308. -
    5308. Another Example, now with a polynomial fit
      5309. -
    5309. To think about, first part
      5310. -
    5310. More thinking
      5311. -
    5311. Still thinking
      5312. -
    5312. What does centering (subtracting the mean values) mean mathematically?
      5313. -
    5313. Further Manipulations
      5314. -
    5314. Wrapping it up
      5315. -
    5315. Linear Regression code, Intercept handling first
      5316. -
    5316. Code Examples
      5317. -
    5317. Taking out the mean
      5318. -
    5318. Friday September 9
      5319. -
    5319. Linking the regression analysis with a statistical interpretation
      5320. -
    5320. Assumptions made
      5321. -
    5321. Expectation value and variance
      5322. -
    5322. Expectation value and variance for \( \boldsymbol{\beta} \)
      5323. -
    5323. Deriving OLS from a probability distribution
      5324. -
    5324. Independent and Identically Distrubuted (iid)
      5325. -
    5325. Maximum Likelihood Estimation (MLE)
      5326. -
    5326. A new Cost Function
      5327. -
    5327. More basic Statistics and Bayes' theorem
      5328. -
    5328. Marginal Probability
      5329. -
    5329. Conditional Probability
      5330. -
    5330. Bayes' Theorem
      5331. -
    5331. Interpretations of Bayes' Theorem
      5332. -
    5332. Example of Usage of Bayes' theorem
      5333. -
    5333. Doing it correctly
      5334. -
    5334. Bayes' Theorem and Ridge and Lasso Regression
      5335. -
    5335. Test Function for what happens with OLS, Ridge and Lasso
      5336. -
    5336. Invoking Bayes' theorem
      5337. -
    5337. Ridge and Bayes
      5338. -
    5338. Lasso and Bayes
      5339. -
    5339. Exercise 1: mean values and variances in linear regression
      5340. -
    5340. Exercise 2: Adding Ridge and Lasso Regression
      5341. +
    5341. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5342. +
    5342. Linear Regression and the SVD
      5343. +
    5343. What does it mean?
      5344. +
    5344. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5345. +
    5345. Code for SVD and Inversion of Matrices
      5346. +
    5346. Inverse of Rectangular Matrix
      5347. +
    5347. Ridge and LASSO Regression
      5348. +
    5348. From OLS to Ridge and Lasso
      5349. +
    5349. Deriving the Ridge Regression Equations
      5350. +
    5350. Note on Scikit-Learn
      5351. +
    5351. Comparison with OLS
      5352. +
    5352. SVD analysis
      5353. +
    5353. Interpreting the Ridge results
      5354. +
    5354. More interpretations
      5355. +
    5355. Deriving the Lasso Regression Equations
      5356. +
    5356. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5357. +
    5357. Ridge Regression
      5358. +
    5358. Lasso Regression
      5359. +
    5359. Yet another Example
      5360. +
    5360. The OLS case
      5361. +
    5361. The Ridge case
      5362. +
    5362. Writing the Cost Function
      5363. +
    5363. Lasso case
      5364. +
    5364. The first Case
      5365. +
    5365. Simple code for solving the above problem
      5366. +
    5366. With Lasso Regression
      5367. +
    5367. Another Example, now with a polynomial fit
      5368. +
    5368. To think about, first part
      5369. +
    5369. More thinking
      5370. +
    5370. Still thinking
      5371. +
    5371. What does centering (subtracting the mean values) mean mathematically?
      5372. +
    5372. Further Manipulations
      5373. +
    5373. Wrapping it up
      5374. +
    5374. Linear Regression code, Intercept handling first
      5375. +
    5375. Code Examples
      5376. +
    5376. Taking out the mean
      5377. +
    5377. Friday September 9
      5378. +
    5378. Linking the regression analysis with a statistical interpretation
      5379. +
    5379. Assumptions made
      5380. +
    5380. Expectation value and variance
      5381. +
    5381. Expectation value and variance for \( \boldsymbol{\beta} \)
      5382. +
    5382. Deriving OLS from a probability distribution
      5383. +
    5383. Independent and Identically Distrubuted (iid)
      5384. +
    5384. Maximum Likelihood Estimation (MLE)
      5385. +
    5385. A new Cost Function
      5386. +
    5386. More basic Statistics and Bayes' theorem
      5387. +
    5387. Marginal Probability
      5388. +
    5388. Conditional Probability
      5389. +
    5389. Bayes' Theorem
      5390. +
    5390. Interpretations of Bayes' Theorem
      5391. +
    5391. Example of Usage of Bayes' theorem
      5392. +
    5392. Doing it correctly
      5393. +
    5393. Bayes' Theorem and Ridge and Lasso Regression
      5394. +
    5394. Test Function for what happens with OLS, Ridge and Lasso
      5395. +
    5395. Invoking Bayes' theorem
      5396. +
    5396. Ridge and Bayes
      5397. +
    5397. Lasso and Bayes
      5398. +
    5398. Exercise 1: mean values and variances in linear regression
      5399. +
    5399. Exercise 2: Adding Ridge and Lasso Regression
      5400. @@ -297,27 +295,38 @@ MathJax.Hub.Config({

       

       

       

      -

      Deriving OLS from a probability distribution

      +

      Independent and Identically Distrubuted (iid)

      -

      Our basic assumption when we derived the OLS equations was to assume -that our output is determined by a given continuous function -\( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal -distribution with zero mean value and an undetermined variance -\( \sigma^2 \). +

      We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. +We define this distribution as

      - -

      We found above that the outputs \( \boldsymbol{y} \) have a mean value given by -\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to -the design matrix are not stochastic variables, we can assume that the -probability distribution of our targets is also a normal distribution -but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a -single output \( y_i \) is given by the Gaussian distribution -

      - $$ -y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}, $$ +

      which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \).

      + +

      Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have

      + +$$ +p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}). +$$ + +

      We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is +in case we have a simple one-dimensional input and output case +

      +$$ +\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})]. +$$ + +

      In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). +We can now rewrite the above probability as +

      +$$ +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +$$ + +

      It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \).

      @@ -344,7 +353,7 @@ $$

    5400. 53
    5401. 54
    5402. ...
      5403. -
    5403. 60
      5404. +
    5404. 59
    5405. »
      5406. diff --git a/doc/pub/week36/html/._week36-bs045.html b/doc/pub/week36/html/._week36-bs045.html index de31e2cd5..e8f2b2d4b 100644 --- a/doc/pub/week36/html/._week36-bs045.html +++ b/doc/pub/week36/html/._week36-bs045.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    5406. Plans for week 36
      5407. -
    5407. Thursday September 8
      5408. -
    5408. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5409. -
    5409. Linear Regression and the SVD
      5410. -
    5410. What does it mean?
      5411. -
    5411. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5412. -
    5412. Code for SVD and Inversion of Matrices
      5413. -
    5413. Inverse of Rectangular Matrix
      5414. -
    5414. Ridge and LASSO Regression
      5415. -
    5415. From OLS to Ridge and Lasso
      5416. -
    5416. Deriving the Ridge Regression Equations
      5417. -
    5417. Note on Scikit-Learn
      5418. -
    5418. Comparison with OLS
      5419. -
    5419. SVD analysis
      5420. -
    5420. Interpreting the Ridge results
      5421. -
    5421. More interpretations
      5422. -
    5422. Deriving the Lasso Regression Equations
      5423. -
    5423. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5424. -
    5424. Ridge Regression
      5425. -
    5425. Lasso Regression
      5426. -
    5426. Yet another Example
      5427. -
    5427. The OLS case
      5428. -
    5428. The Ridge case
      5429. -
    5429. Writing the Cost Function
      5430. -
    5430. Lasso case
      5431. -
    5431. The first Case
      5432. -
    5432. Simple code for solving the above problem
      5433. -
    5433. With Lasso Regression
      5434. -
    5434. Another Example, now with a polynomial fit
      5435. -
    5435. To think about, first part
      5436. -
    5436. More thinking
      5437. -
    5437. Still thinking
      5438. -
    5438. What does centering (subtracting the mean values) mean mathematically?
      5439. -
    5439. Further Manipulations
      5440. -
    5440. Wrapping it up
      5441. -
    5441. Linear Regression code, Intercept handling first
      5442. -
    5442. Code Examples
      5443. -
    5443. Taking out the mean
      5444. -
    5444. Friday September 9
      5445. -
    5445. Linking the regression analysis with a statistical interpretation
      5446. -
    5446. Assumptions made
      5447. -
    5447. Expectation value and variance
      5448. -
    5448. Expectation value and variance for \( \boldsymbol{\beta} \)
      5449. -
    5449. Deriving OLS from a probability distribution
      5450. -
    5450. Independent and Identically Distrubuted (iid)
      5451. -
    5451. Maximum Likelihood Estimation (MLE)
      5452. -
    5452. A new Cost Function
      5453. -
    5453. More basic Statistics and Bayes' theorem
      5454. -
    5454. Marginal Probability
      5455. -
    5455. Conditional Probability
      5456. -
    5456. Bayes' Theorem
      5457. -
    5457. Interpretations of Bayes' Theorem
      5458. -
    5458. Example of Usage of Bayes' theorem
      5459. -
    5459. Doing it correctly
      5460. -
    5460. Bayes' Theorem and Ridge and Lasso Regression
      5461. -
    5461. Test Function for what happens with OLS, Ridge and Lasso
      5462. -
    5462. Invoking Bayes' theorem
      5463. -
    5463. Ridge and Bayes
      5464. -
    5464. Lasso and Bayes
      5465. -
    5465. Exercise 1: mean values and variances in linear regression
      5466. -
    5466. Exercise 2: Adding Ridge and Lasso Regression
      5467. +
    5467. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5468. +
    5468. Linear Regression and the SVD
      5469. +
    5469. What does it mean?
      5470. +
    5470. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5471. +
    5471. Code for SVD and Inversion of Matrices
      5472. +
    5472. Inverse of Rectangular Matrix
      5473. +
    5473. Ridge and LASSO Regression
      5474. +
    5474. From OLS to Ridge and Lasso
      5475. +
    5475. Deriving the Ridge Regression Equations
      5476. +
    5476. Note on Scikit-Learn
      5477. +
    5477. Comparison with OLS
      5478. +
    5478. SVD analysis
      5479. +
    5479. Interpreting the Ridge results
      5480. +
    5480. More interpretations
      5481. +
    5481. Deriving the Lasso Regression Equations
      5482. +
    5482. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5483. +
    5483. Ridge Regression
      5484. +
    5484. Lasso Regression
      5485. +
    5485. Yet another Example
      5486. +
    5486. The OLS case
      5487. +
    5487. The Ridge case
      5488. +
    5488. Writing the Cost Function
      5489. +
    5489. Lasso case
      5490. +
    5490. The first Case
      5491. +
    5491. Simple code for solving the above problem
      5492. +
    5492. With Lasso Regression
      5493. +
    5493. Another Example, now with a polynomial fit
      5494. +
    5494. To think about, first part
      5495. +
    5495. More thinking
      5496. +
    5496. Still thinking
      5497. +
    5497. What does centering (subtracting the mean values) mean mathematically?
      5498. +
    5498. Further Manipulations
      5499. +
    5499. Wrapping it up
      5500. +
    5500. Linear Regression code, Intercept handling first
      5501. +
    5501. Code Examples
      5502. +
    5502. Taking out the mean
      5503. +
    5503. Friday September 9
      5504. +
    5504. Linking the regression analysis with a statistical interpretation
      5505. +
    5505. Assumptions made
      5506. +
    5506. Expectation value and variance
      5507. +
    5507. Expectation value and variance for \( \boldsymbol{\beta} \)
      5508. +
    5508. Deriving OLS from a probability distribution
      5509. +
    5509. Independent and Identically Distrubuted (iid)
      5510. +
    5510. Maximum Likelihood Estimation (MLE)
      5511. +
    5511. A new Cost Function
      5512. +
    5512. More basic Statistics and Bayes' theorem
      5513. +
    5513. Marginal Probability
      5514. +
    5514. Conditional Probability
      5515. +
    5515. Bayes' Theorem
      5516. +
    5516. Interpretations of Bayes' Theorem
      5517. +
    5517. Example of Usage of Bayes' theorem
      5518. +
    5518. Doing it correctly
      5519. +
    5519. Bayes' Theorem and Ridge and Lasso Regression
      5520. +
    5520. Test Function for what happens with OLS, Ridge and Lasso
      5521. +
    5521. Invoking Bayes' theorem
      5522. +
    5522. Ridge and Bayes
      5523. +
    5523. Lasso and Bayes
      5524. +
    5524. Exercise 1: mean values and variances in linear regression
      5525. +
    5525. Exercise 2: Adding Ridge and Lasso Regression
      5526. @@ -297,38 +295,32 @@ MathJax.Hub.Config({

       

       

       

      -

      Independent and Identically Distrubuted (iid)

      +

      Maximum Likelihood Estimation (MLE)

      -

      We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. -We define this distribution as +

      In statistics, maximum likelihood estimation (MLE) is a method of +estimating the parameters of an assumed probability distribution, +given some observed data. This is achieved by maximizing a likelihood +function so that, under the assumed statistical model, the observed +data is the most probable.

      -$$ -p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}, -$$ -

      which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \).

      - -

      Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have

      - -$$ -p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}). -$$ - -

      We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is -in case we have a simple one-dimensional input and output case +

      We will assume here that our events are given by the above Gaussian +distribution and we will determine the optimal parameters \( \beta \) by +maximizing the above PDF. However, computing the derivatives of a +product function is cumbersome and can easily lead to overflow and/or +underflowproblems, with potentials for loss of numerical precision.

      -$$ -\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})]. -$$ -

      In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). -We can now rewrite the above probability as +

      In practice, it is more convenient to maximize the logarithm of the +PDF because it is a monotonically increasing function of the argument. +Alternatively, and this will be our option, we will minimize the +negative of the logarithm since this is a monotonically decreasing +function.

      -$$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. -$$ -

      It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \).

      +

      Note also that maximization/minimization of the logarithm of the PDF +is equivalent to the maximization/minimization of the function itself. +

      @@ -355,7 +347,7 @@ $$

    5526. 54
    5527. 55
    5528. ...
      5529. -
    5529. 60
      5530. +
    5530. 59
    5531. »
      5532. diff --git a/doc/pub/week36/html/._week36-bs046.html b/doc/pub/week36/html/._week36-bs046.html index cd6ecbe89..11e698ad0 100644 --- a/doc/pub/week36/html/._week36-bs046.html +++ b/doc/pub/week36/html/._week36-bs046.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    5532. Plans for week 36
      5533. -
    5533. Thursday September 8
      5534. -
    5534. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5535. -
    5535. Linear Regression and the SVD
      5536. -
    5536. What does it mean?
      5537. -
    5537. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5538. -
    5538. Code for SVD and Inversion of Matrices
      5539. -
    5539. Inverse of Rectangular Matrix
      5540. -
    5540. Ridge and LASSO Regression
      5541. -
    5541. From OLS to Ridge and Lasso
      5542. -
    5542. Deriving the Ridge Regression Equations
      5543. -
    5543. Note on Scikit-Learn
      5544. -
    5544. Comparison with OLS
      5545. -
    5545. SVD analysis
      5546. -
    5546. Interpreting the Ridge results
      5547. -
    5547. More interpretations
      5548. -
    5548. Deriving the Lasso Regression Equations
      5549. -
    5549. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5550. -
    5550. Ridge Regression
      5551. -
    5551. Lasso Regression
      5552. -
    5552. Yet another Example
      5553. -
    5553. The OLS case
      5554. -
    5554. The Ridge case
      5555. -
    5555. Writing the Cost Function
      5556. -
    5556. Lasso case
      5557. -
    5557. The first Case
      5558. -
    5558. Simple code for solving the above problem
      5559. -
    5559. With Lasso Regression
      5560. -
    5560. Another Example, now with a polynomial fit
      5561. -
    5561. To think about, first part
      5562. -
    5562. More thinking
      5563. -
    5563. Still thinking
      5564. -
    5564. What does centering (subtracting the mean values) mean mathematically?
      5565. -
    5565. Further Manipulations
      5566. -
    5566. Wrapping it up
      5567. -
    5567. Linear Regression code, Intercept handling first
      5568. -
    5568. Code Examples
      5569. -
    5569. Taking out the mean
      5570. -
    5570. Friday September 9
      5571. -
    5571. Linking the regression analysis with a statistical interpretation
      5572. -
    5572. Assumptions made
      5573. -
    5573. Expectation value and variance
      5574. -
    5574. Expectation value and variance for \( \boldsymbol{\beta} \)
      5575. -
    5575. Deriving OLS from a probability distribution
      5576. -
    5576. Independent and Identically Distrubuted (iid)
      5577. -
    5577. Maximum Likelihood Estimation (MLE)
      5578. -
    5578. A new Cost Function
      5579. -
    5579. More basic Statistics and Bayes' theorem
      5580. -
    5580. Marginal Probability
      5581. -
    5581. Conditional Probability
      5582. -
    5582. Bayes' Theorem
      5583. -
    5583. Interpretations of Bayes' Theorem
      5584. -
    5584. Example of Usage of Bayes' theorem
      5585. -
    5585. Doing it correctly
      5586. -
    5586. Bayes' Theorem and Ridge and Lasso Regression
      5587. -
    5587. Test Function for what happens with OLS, Ridge and Lasso
      5588. -
    5588. Invoking Bayes' theorem
      5589. -
    5589. Ridge and Bayes
      5590. -
    5590. Lasso and Bayes
      5591. -
    5591. Exercise 1: mean values and variances in linear regression
      5592. -
    5592. Exercise 2: Adding Ridge and Lasso Regression
      5593. +
    5593. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5594. +
    5594. Linear Regression and the SVD
      5595. +
    5595. What does it mean?
      5596. +
    5596. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5597. +
    5597. Code for SVD and Inversion of Matrices
      5598. +
    5598. Inverse of Rectangular Matrix
      5599. +
    5599. Ridge and LASSO Regression
      5600. +
    5600. From OLS to Ridge and Lasso
      5601. +
    5601. Deriving the Ridge Regression Equations
      5602. +
    5602. Note on Scikit-Learn
      5603. +
    5603. Comparison with OLS
      5604. +
    5604. SVD analysis
      5605. +
    5605. Interpreting the Ridge results
      5606. +
    5606. More interpretations
      5607. +
    5607. Deriving the Lasso Regression Equations
      5608. +
    5608. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5609. +
    5609. Ridge Regression
      5610. +
    5610. Lasso Regression
      5611. +
    5611. Yet another Example
      5612. +
    5612. The OLS case
      5613. +
    5613. The Ridge case
      5614. +
    5614. Writing the Cost Function
      5615. +
    5615. Lasso case
      5616. +
    5616. The first Case
      5617. +
    5617. Simple code for solving the above problem
      5618. +
    5618. With Lasso Regression
      5619. +
    5619. Another Example, now with a polynomial fit
      5620. +
    5620. To think about, first part
      5621. +
    5621. More thinking
      5622. +
    5622. Still thinking
      5623. +
    5623. What does centering (subtracting the mean values) mean mathematically?
      5624. +
    5624. Further Manipulations
      5625. +
    5625. Wrapping it up
      5626. +
    5626. Linear Regression code, Intercept handling first
      5627. +
    5627. Code Examples
      5628. +
    5628. Taking out the mean
      5629. +
    5629. Friday September 9
      5630. +
    5630. Linking the regression analysis with a statistical interpretation
      5631. +
    5631. Assumptions made
      5632. +
    5632. Expectation value and variance
      5633. +
    5633. Expectation value and variance for \( \boldsymbol{\beta} \)
      5634. +
    5634. Deriving OLS from a probability distribution
      5635. +
    5635. Independent and Identically Distrubuted (iid)
      5636. +
    5636. Maximum Likelihood Estimation (MLE)
      5637. +
    5637. A new Cost Function
      5638. +
    5638. More basic Statistics and Bayes' theorem
      5639. +
    5639. Marginal Probability
      5640. +
    5640. Conditional Probability
      5641. +
    5641. Bayes' Theorem
      5642. +
    5642. Interpretations of Bayes' Theorem
      5643. +
    5643. Example of Usage of Bayes' theorem
      5644. +
    5644. Doing it correctly
      5645. +
    5645. Bayes' Theorem and Ridge and Lasso Regression
      5646. +
    5646. Test Function for what happens with OLS, Ridge and Lasso
      5647. +
    5647. Invoking Bayes' theorem
      5648. +
    5648. Ridge and Bayes
      5649. +
    5649. Lasso and Bayes
      5650. +
    5650. Exercise 1: mean values and variances in linear regression
      5651. +
    5651. Exercise 2: Adding Ridge and Lasso Regression
      5652. @@ -297,32 +295,31 @@ MathJax.Hub.Config({

       

       

       

      -

      Maximum Likelihood Estimation (MLE)

      +

      A new Cost Function

      -

      In statistics, maximum likelihood estimation (MLE) is a method of -estimating the parameters of an assumed probability distribution, -given some observed data. This is achieved by maximizing a likelihood -function so that, under the assumed statistical model, the observed -data is the most probable. -

      +

      We could now define a new cost function to minimize, namely the negative logarithm of the above PDF

      -

      We will assume here that our events are given by the above Gaussian -distribution and we will determine the optimal parameters \( \beta \) by -maximizing the above PDF. However, computing the derivatives of a -product function is cumbersome and can easily lead to overflow and/or -underflowproblems, with potentials for loss of numerical precision. -

      +$$ +C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}, +$$ -

      In practice, it is more convenient to maximize the logarithm of the -PDF because it is a monotonically increasing function of the argument. -Alternatively, and this will be our option, we will minimize the -negative of the logarithm since this is a monotonically decreasing -function. -

      +

      which becomes

      +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}. +$$ -

      Note also that maximization/minimization of the logarithm of the PDF -is equivalent to the maximization/minimization of the function itself. -

      +

      Taking the derivative of the new cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely

      + +$$ +\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, +$$ + +

      which leads to the well-known OLS equation for the optimal paramters \( \beta \)

      +$$ +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! +$$ + +

      Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics.

      @@ -349,7 +346,7 @@ is equivalent to the maximization/minimization of the function itself.

    5652. 55
    5653. 56
    5654. ...
      5655. -
    5655. 60
      5656. +
    5656. 59
    5657. »
      5658. diff --git a/doc/pub/week36/html/._week36-bs047.html b/doc/pub/week36/html/._week36-bs047.html index 298886237..c91b1e9db 100644 --- a/doc/pub/week36/html/._week36-bs047.html +++ b/doc/pub/week36/html/._week36-bs047.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    5658. Plans for week 36
      5659. -
    5659. Thursday September 8
      5660. -
    5660. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5661. -
    5661. Linear Regression and the SVD
      5662. -
    5662. What does it mean?
      5663. -
    5663. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5664. -
    5664. Code for SVD and Inversion of Matrices
      5665. -
    5665. Inverse of Rectangular Matrix
      5666. -
    5666. Ridge and LASSO Regression
      5667. -
    5667. From OLS to Ridge and Lasso
      5668. -
    5668. Deriving the Ridge Regression Equations
      5669. -
    5669. Note on Scikit-Learn
      5670. -
    5670. Comparison with OLS
      5671. -
    5671. SVD analysis
      5672. -
    5672. Interpreting the Ridge results
      5673. -
    5673. More interpretations
      5674. -
    5674. Deriving the Lasso Regression Equations
      5675. -
    5675. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5676. -
    5676. Ridge Regression
      5677. -
    5677. Lasso Regression
      5678. -
    5678. Yet another Example
      5679. -
    5679. The OLS case
      5680. -
    5680. The Ridge case
      5681. -
    5681. Writing the Cost Function
      5682. -
    5682. Lasso case
      5683. -
    5683. The first Case
      5684. -
    5684. Simple code for solving the above problem
      5685. -
    5685. With Lasso Regression
      5686. -
    5686. Another Example, now with a polynomial fit
      5687. -
    5687. To think about, first part
      5688. -
    5688. More thinking
      5689. -
    5689. Still thinking
      5690. -
    5690. What does centering (subtracting the mean values) mean mathematically?
      5691. -
    5691. Further Manipulations
      5692. -
    5692. Wrapping it up
      5693. -
    5693. Linear Regression code, Intercept handling first
      5694. -
    5694. Code Examples
      5695. -
    5695. Taking out the mean
      5696. -
    5696. Friday September 9
      5697. -
    5697. Linking the regression analysis with a statistical interpretation
      5698. -
    5698. Assumptions made
      5699. -
    5699. Expectation value and variance
      5700. -
    5700. Expectation value and variance for \( \boldsymbol{\beta} \)
      5701. -
    5701. Deriving OLS from a probability distribution
      5702. -
    5702. Independent and Identically Distrubuted (iid)
      5703. -
    5703. Maximum Likelihood Estimation (MLE)
      5704. -
    5704. A new Cost Function
      5705. -
    5705. More basic Statistics and Bayes' theorem
      5706. -
    5706. Marginal Probability
      5707. -
    5707. Conditional Probability
      5708. -
    5708. Bayes' Theorem
      5709. -
    5709. Interpretations of Bayes' Theorem
      5710. -
    5710. Example of Usage of Bayes' theorem
      5711. -
    5711. Doing it correctly
      5712. -
    5712. Bayes' Theorem and Ridge and Lasso Regression
      5713. -
    5713. Test Function for what happens with OLS, Ridge and Lasso
      5714. -
    5714. Invoking Bayes' theorem
      5715. -
    5715. Ridge and Bayes
      5716. -
    5716. Lasso and Bayes
      5717. -
    5717. Exercise 1: mean values and variances in linear regression
      5718. -
    5718. Exercise 2: Adding Ridge and Lasso Regression
      5719. +
    5719. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5720. +
    5720. Linear Regression and the SVD
      5721. +
    5721. What does it mean?
      5722. +
    5722. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5723. +
    5723. Code for SVD and Inversion of Matrices
      5724. +
    5724. Inverse of Rectangular Matrix
      5725. +
    5725. Ridge and LASSO Regression
      5726. +
    5726. From OLS to Ridge and Lasso
      5727. +
    5727. Deriving the Ridge Regression Equations
      5728. +
    5728. Note on Scikit-Learn
      5729. +
    5729. Comparison with OLS
      5730. +
    5730. SVD analysis
      5731. +
    5731. Interpreting the Ridge results
      5732. +
    5732. More interpretations
      5733. +
    5733. Deriving the Lasso Regression Equations
      5734. +
    5734. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5735. +
    5735. Ridge Regression
      5736. +
    5736. Lasso Regression
      5737. +
    5737. Yet another Example
      5738. +
    5738. The OLS case
      5739. +
    5739. The Ridge case
      5740. +
    5740. Writing the Cost Function
      5741. +
    5741. Lasso case
      5742. +
    5742. The first Case
      5743. +
    5743. Simple code for solving the above problem
      5744. +
    5744. With Lasso Regression
      5745. +
    5745. Another Example, now with a polynomial fit
      5746. +
    5746. To think about, first part
      5747. +
    5747. More thinking
      5748. +
    5748. Still thinking
      5749. +
    5749. What does centering (subtracting the mean values) mean mathematically?
      5750. +
    5750. Further Manipulations
      5751. +
    5751. Wrapping it up
      5752. +
    5752. Linear Regression code, Intercept handling first
      5753. +
    5753. Code Examples
      5754. +
    5754. Taking out the mean
      5755. +
    5755. Friday September 9
      5756. +
    5756. Linking the regression analysis with a statistical interpretation
      5757. +
    5757. Assumptions made
      5758. +
    5758. Expectation value and variance
      5759. +
    5759. Expectation value and variance for \( \boldsymbol{\beta} \)
      5760. +
    5760. Deriving OLS from a probability distribution
      5761. +
    5761. Independent and Identically Distrubuted (iid)
      5762. +
    5762. Maximum Likelihood Estimation (MLE)
      5763. +
    5763. A new Cost Function
      5764. +
    5764. More basic Statistics and Bayes' theorem
      5765. +
    5765. Marginal Probability
      5766. +
    5766. Conditional Probability
      5767. +
    5767. Bayes' Theorem
      5768. +
    5768. Interpretations of Bayes' Theorem
      5769. +
    5769. Example of Usage of Bayes' theorem
      5770. +
    5770. Doing it correctly
      5771. +
    5771. Bayes' Theorem and Ridge and Lasso Regression
      5772. +
    5772. Test Function for what happens with OLS, Ridge and Lasso
      5773. +
    5773. Invoking Bayes' theorem
      5774. +
    5774. Ridge and Bayes
      5775. +
    5775. Lasso and Bayes
      5776. +
    5776. Exercise 1: mean values and variances in linear regression
      5777. +
    5777. Exercise 2: Adding Ridge and Lasso Regression
      5778. @@ -297,31 +295,41 @@ MathJax.Hub.Config({

       

       

       

      -

      A new Cost Function

      +

      More basic Statistics and Bayes' theorem

      -

      We could now define a new cost function to minimize, namely the negative logarithm of the above PDF

      +

      A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. +

      +

      Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \).

      + +

      We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. +The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). +

      + +
      +
      + $$ -C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}, +p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). +$$ +
      +
      + + +
      +
      + +$$ +p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X), $$ -

      which becomes

      -$$ -C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}. -$$ +

      where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \).

      +
      +
      -

      Taking the derivative of the new cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely

      -$$ -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, -$$ - -

      which leads to the well-known OLS equation for the optimal paramters \( \beta \)

      -$$ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! -$$ - -

      Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics.

      +

      If we have independent events then \( p(X,Y)=p(X)p(Y) \).

      @@ -348,7 +356,7 @@ $$

    5778. 56
    5779. 57
    5780. ...
      5781. -
    5781. 60
      5782. +
    5782. 59
    5783. »
      5784. diff --git a/doc/pub/week36/html/._week36-bs048.html b/doc/pub/week36/html/._week36-bs048.html index 752f71e51..d4253959f 100644 --- a/doc/pub/week36/html/._week36-bs048.html +++ b/doc/pub/week36/html/._week36-bs048.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    5784. Plans for week 36
      5785. -
    5785. Thursday September 8
      5786. -
    5786. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5787. -
    5787. Linear Regression and the SVD
      5788. -
    5788. What does it mean?
      5789. -
    5789. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5790. -
    5790. Code for SVD and Inversion of Matrices
      5791. -
    5791. Inverse of Rectangular Matrix
      5792. -
    5792. Ridge and LASSO Regression
      5793. -
    5793. From OLS to Ridge and Lasso
      5794. -
    5794. Deriving the Ridge Regression Equations
      5795. -
    5795. Note on Scikit-Learn
      5796. -
    5796. Comparison with OLS
      5797. -
    5797. SVD analysis
      5798. -
    5798. Interpreting the Ridge results
      5799. -
    5799. More interpretations
      5800. -
    5800. Deriving the Lasso Regression Equations
      5801. -
    5801. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5802. -
    5802. Ridge Regression
      5803. -
    5803. Lasso Regression
      5804. -
    5804. Yet another Example
      5805. -
    5805. The OLS case
      5806. -
    5806. The Ridge case
      5807. -
    5807. Writing the Cost Function
      5808. -
    5808. Lasso case
      5809. -
    5809. The first Case
      5810. -
    5810. Simple code for solving the above problem
      5811. -
    5811. With Lasso Regression
      5812. -
    5812. Another Example, now with a polynomial fit
      5813. -
    5813. To think about, first part
      5814. -
    5814. More thinking
      5815. -
    5815. Still thinking
      5816. -
    5816. What does centering (subtracting the mean values) mean mathematically?
      5817. -
    5817. Further Manipulations
      5818. -
    5818. Wrapping it up
      5819. -
    5819. Linear Regression code, Intercept handling first
      5820. -
    5820. Code Examples
      5821. -
    5821. Taking out the mean
      5822. -
    5822. Friday September 9
      5823. -
    5823. Linking the regression analysis with a statistical interpretation
      5824. -
    5824. Assumptions made
      5825. -
    5825. Expectation value and variance
      5826. -
    5826. Expectation value and variance for \( \boldsymbol{\beta} \)
      5827. -
    5827. Deriving OLS from a probability distribution
      5828. -
    5828. Independent and Identically Distrubuted (iid)
      5829. -
    5829. Maximum Likelihood Estimation (MLE)
      5830. -
    5830. A new Cost Function
      5831. -
    5831. More basic Statistics and Bayes' theorem
      5832. -
    5832. Marginal Probability
      5833. -
    5833. Conditional Probability
      5834. -
    5834. Bayes' Theorem
      5835. -
    5835. Interpretations of Bayes' Theorem
      5836. -
    5836. Example of Usage of Bayes' theorem
      5837. -
    5837. Doing it correctly
      5838. -
    5838. Bayes' Theorem and Ridge and Lasso Regression
      5839. -
    5839. Test Function for what happens with OLS, Ridge and Lasso
      5840. -
    5840. Invoking Bayes' theorem
      5841. -
    5841. Ridge and Bayes
      5842. -
    5842. Lasso and Bayes
      5843. -
    5843. Exercise 1: mean values and variances in linear regression
      5844. -
    5844. Exercise 2: Adding Ridge and Lasso Regression
      5845. +
    5845. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5846. +
    5846. Linear Regression and the SVD
      5847. +
    5847. What does it mean?
      5848. +
    5848. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5849. +
    5849. Code for SVD and Inversion of Matrices
      5850. +
    5850. Inverse of Rectangular Matrix
      5851. +
    5851. Ridge and LASSO Regression
      5852. +
    5852. From OLS to Ridge and Lasso
      5853. +
    5853. Deriving the Ridge Regression Equations
      5854. +
    5854. Note on Scikit-Learn
      5855. +
    5855. Comparison with OLS
      5856. +
    5856. SVD analysis
      5857. +
    5857. Interpreting the Ridge results
      5858. +
    5858. More interpretations
      5859. +
    5859. Deriving the Lasso Regression Equations
      5860. +
    5860. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5861. +
    5861. Ridge Regression
      5862. +
    5862. Lasso Regression
      5863. +
    5863. Yet another Example
      5864. +
    5864. The OLS case
      5865. +
    5865. The Ridge case
      5866. +
    5866. Writing the Cost Function
      5867. +
    5867. Lasso case
      5868. +
    5868. The first Case
      5869. +
    5869. Simple code for solving the above problem
      5870. +
    5870. With Lasso Regression
      5871. +
    5871. Another Example, now with a polynomial fit
      5872. +
    5872. To think about, first part
      5873. +
    5873. More thinking
      5874. +
    5874. Still thinking
      5875. +
    5875. What does centering (subtracting the mean values) mean mathematically?
      5876. +
    5876. Further Manipulations
      5877. +
    5877. Wrapping it up
      5878. +
    5878. Linear Regression code, Intercept handling first
      5879. +
    5879. Code Examples
      5880. +
    5880. Taking out the mean
      5881. +
    5881. Friday September 9
      5882. +
    5882. Linking the regression analysis with a statistical interpretation
      5883. +
    5883. Assumptions made
      5884. +
    5884. Expectation value and variance
      5885. +
    5885. Expectation value and variance for \( \boldsymbol{\beta} \)
      5886. +
    5886. Deriving OLS from a probability distribution
      5887. +
    5887. Independent and Identically Distrubuted (iid)
      5888. +
    5888. Maximum Likelihood Estimation (MLE)
      5889. +
    5889. A new Cost Function
      5890. +
    5890. More basic Statistics and Bayes' theorem
      5891. +
    5891. Marginal Probability
      5892. +
    5892. Conditional Probability
      5893. +
    5893. Bayes' Theorem
      5894. +
    5894. Interpretations of Bayes' Theorem
      5895. +
    5895. Example of Usage of Bayes' theorem
      5896. +
    5896. Doing it correctly
      5897. +
    5897. Bayes' Theorem and Ridge and Lasso Regression
      5898. +
    5898. Test Function for what happens with OLS, Ridge and Lasso
      5899. +
    5899. Invoking Bayes' theorem
      5900. +
    5900. Ridge and Bayes
      5901. +
    5901. Lasso and Bayes
      5902. +
    5902. Exercise 1: mean values and variances in linear regression
      5903. +
    5903. Exercise 2: Adding Ridge and Lasso Regression
      5904. @@ -297,42 +295,19 @@ MathJax.Hub.Config({

       

       

       

      -

      More basic Statistics and Bayes' theorem

      - -

      A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. -Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. -

      - -

      Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \).

      - -

      We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. -The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). -

      +

      Marginal Probability

      +

      The marginal probability is defined in terms of only one of the set of variables \( X,Y \). For a discrete probability we have

      $$ -p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). +p(X)=\sum_{i=0}^{n-1}p(X,Y=y_i)=\sum_{i=0}^{n-1}p(X\vert Y=y_i)p(Y=y_i)=\sum_{i=0}^{n-1}p(X\vert y_i)p(y_i). $$
      -
      -
      - -$$ -p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X), -$$ - -

      where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \).

      -
      -
      - - -

      If we have independent events then \( p(X,Y)=p(X)p(Y) \).

      -

      diff --git a/doc/pub/week36/html/._week36-bs049.html b/doc/pub/week36/html/._week36-bs049.html index c20dafc4c..66f4abff1 100644 --- a/doc/pub/week36/html/._week36-bs049.html +++ b/doc/pub/week36/html/._week36-bs049.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    5904. Plans for week 36
      5905. -
    5905. Thursday September 8
      5906. -
    5906. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5907. -
    5907. Linear Regression and the SVD
      5908. -
    5908. What does it mean?
      5909. -
    5909. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5910. -
    5910. Code for SVD and Inversion of Matrices
      5911. -
    5911. Inverse of Rectangular Matrix
      5912. -
    5912. Ridge and LASSO Regression
      5913. -
    5913. From OLS to Ridge and Lasso
      5914. -
    5914. Deriving the Ridge Regression Equations
      5915. -
    5915. Note on Scikit-Learn
      5916. -
    5916. Comparison with OLS
      5917. -
    5917. SVD analysis
      5918. -
    5918. Interpreting the Ridge results
      5919. -
    5919. More interpretations
      5920. -
    5920. Deriving the Lasso Regression Equations
      5921. -
    5921. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5922. -
    5922. Ridge Regression
      5923. -
    5923. Lasso Regression
      5924. -
    5924. Yet another Example
      5925. -
    5925. The OLS case
      5926. -
    5926. The Ridge case
      5927. -
    5927. Writing the Cost Function
      5928. -
    5928. Lasso case
      5929. -
    5929. The first Case
      5930. -
    5930. Simple code for solving the above problem
      5931. -
    5931. With Lasso Regression
      5932. -
    5932. Another Example, now with a polynomial fit
      5933. -
    5933. To think about, first part
      5934. -
    5934. More thinking
      5935. -
    5935. Still thinking
      5936. -
    5936. What does centering (subtracting the mean values) mean mathematically?
      5937. -
    5937. Further Manipulations
      5938. -
    5938. Wrapping it up
      5939. -
    5939. Linear Regression code, Intercept handling first
      5940. -
    5940. Code Examples
      5941. -
    5941. Taking out the mean
      5942. -
    5942. Friday September 9
      5943. -
    5943. Linking the regression analysis with a statistical interpretation
      5944. -
    5944. Assumptions made
      5945. -
    5945. Expectation value and variance
      5946. -
    5946. Expectation value and variance for \( \boldsymbol{\beta} \)
      5947. -
    5947. Deriving OLS from a probability distribution
      5948. -
    5948. Independent and Identically Distrubuted (iid)
      5949. -
    5949. Maximum Likelihood Estimation (MLE)
      5950. -
    5950. A new Cost Function
      5951. -
    5951. More basic Statistics and Bayes' theorem
      5952. -
    5952. Marginal Probability
      5953. -
    5953. Conditional Probability
      5954. -
    5954. Bayes' Theorem
      5955. -
    5955. Interpretations of Bayes' Theorem
      5956. -
    5956. Example of Usage of Bayes' theorem
      5957. -
    5957. Doing it correctly
      5958. -
    5958. Bayes' Theorem and Ridge and Lasso Regression
      5959. -
    5959. Test Function for what happens with OLS, Ridge and Lasso
      5960. -
    5960. Invoking Bayes' theorem
      5961. -
    5961. Ridge and Bayes
      5962. -
    5962. Lasso and Bayes
      5963. -
    5963. Exercise 1: mean values and variances in linear regression
      5964. -
    5964. Exercise 2: Adding Ridge and Lasso Regression
      5965. +
    5965. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      5966. +
    5966. Linear Regression and the SVD
      5967. +
    5967. What does it mean?
      5968. +
    5968. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      5969. +
    5969. Code for SVD and Inversion of Matrices
      5970. +
    5970. Inverse of Rectangular Matrix
      5971. +
    5971. Ridge and LASSO Regression
      5972. +
    5972. From OLS to Ridge and Lasso
      5973. +
    5973. Deriving the Ridge Regression Equations
      5974. +
    5974. Note on Scikit-Learn
      5975. +
    5975. Comparison with OLS
      5976. +
    5976. SVD analysis
      5977. +
    5977. Interpreting the Ridge results
      5978. +
    5978. More interpretations
      5979. +
    5979. Deriving the Lasso Regression Equations
      5980. +
    5980. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      5981. +
    5981. Ridge Regression
      5982. +
    5982. Lasso Regression
      5983. +
    5983. Yet another Example
      5984. +
    5984. The OLS case
      5985. +
    5985. The Ridge case
      5986. +
    5986. Writing the Cost Function
      5987. +
    5987. Lasso case
      5988. +
    5988. The first Case
      5989. +
    5989. Simple code for solving the above problem
      5990. +
    5990. With Lasso Regression
      5991. +
    5991. Another Example, now with a polynomial fit
      5992. +
    5992. To think about, first part
      5993. +
    5993. More thinking
      5994. +
    5994. Still thinking
      5995. +
    5995. What does centering (subtracting the mean values) mean mathematically?
      5996. +
    5996. Further Manipulations
      5997. +
    5997. Wrapping it up
      5998. +
    5998. Linear Regression code, Intercept handling first
      5999. +
    5999. Code Examples
      6000. +
    6000. Taking out the mean
      6001. +
    6001. Friday September 9
      6002. +
    6002. Linking the regression analysis with a statistical interpretation
      6003. +
    6003. Assumptions made
      6004. +
    6004. Expectation value and variance
      6005. +
    6005. Expectation value and variance for \( \boldsymbol{\beta} \)
      6006. +
    6006. Deriving OLS from a probability distribution
      6007. +
    6007. Independent and Identically Distrubuted (iid)
      6008. +
    6008. Maximum Likelihood Estimation (MLE)
      6009. +
    6009. A new Cost Function
      6010. +
    6010. More basic Statistics and Bayes' theorem
      6011. +
    6011. Marginal Probability
      6012. +
    6012. Conditional Probability
      6013. +
    6013. Bayes' Theorem
      6014. +
    6014. Interpretations of Bayes' Theorem
      6015. +
    6015. Example of Usage of Bayes' theorem
      6016. +
    6016. Doing it correctly
      6017. +
    6017. Bayes' Theorem and Ridge and Lasso Regression
      6018. +
    6018. Test Function for what happens with OLS, Ridge and Lasso
      6019. +
    6019. Invoking Bayes' theorem
      6020. +
    6020. Ridge and Bayes
      6021. +
    6021. Lasso and Bayes
      6022. +
    6022. Exercise 1: mean values and variances in linear regression
      6023. +
    6023. Exercise 2: Adding Ridge and Lasso Regression
      6024. @@ -297,14 +295,14 @@ MathJax.Hub.Config({

       

       

       

      -

      Marginal Probability

      +

      Conditional Probability

      -

      The marginal probability is defined in terms of only one of the set of variables \( X,Y \). For a discrete probability we have

      +

      The conditional probability, if \( p(Y) > 0 \), is

      $$ -p(X)=\sum_{i=0}^{n-1}p(X,Y=y_i)=\sum_{i=0}^{n-1}p(X\vert Y=y_i)p(Y=y_i)=\sum_{i=0}^{n-1}p(X\vert y_i)p(y_i). +p(X\vert Y)= \frac{p(X,Y)}{p(Y)}=\frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}. $$
      @@ -334,8 +332,6 @@ $$
    6024. 57
    6025. 58
    6026. 59
      6027. -
    6027. ...
      6028. -
    6028. 60
    6029. »
      6030. diff --git a/doc/pub/week36/html/._week36-bs050.html b/doc/pub/week36/html/._week36-bs050.html index 0065c56ef..fd8c055d7 100644 --- a/doc/pub/week36/html/._week36-bs050.html +++ b/doc/pub/week36/html/._week36-bs050.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    6030. Plans for week 36
      6031. -
    6031. Thursday September 8
      6032. -
    6032. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6033. -
    6033. Linear Regression and the SVD
      6034. -
    6034. What does it mean?
      6035. -
    6035. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6036. -
    6036. Code for SVD and Inversion of Matrices
      6037. -
    6037. Inverse of Rectangular Matrix
      6038. -
    6038. Ridge and LASSO Regression
      6039. -
    6039. From OLS to Ridge and Lasso
      6040. -
    6040. Deriving the Ridge Regression Equations
      6041. -
    6041. Note on Scikit-Learn
      6042. -
    6042. Comparison with OLS
      6043. -
    6043. SVD analysis
      6044. -
    6044. Interpreting the Ridge results
      6045. -
    6045. More interpretations
      6046. -
    6046. Deriving the Lasso Regression Equations
      6047. -
    6047. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6048. -
    6048. Ridge Regression
      6049. -
    6049. Lasso Regression
      6050. -
    6050. Yet another Example
      6051. -
    6051. The OLS case
      6052. -
    6052. The Ridge case
      6053. -
    6053. Writing the Cost Function
      6054. -
    6054. Lasso case
      6055. -
    6055. The first Case
      6056. -
    6056. Simple code for solving the above problem
      6057. -
    6057. With Lasso Regression
      6058. -
    6058. Another Example, now with a polynomial fit
      6059. -
    6059. To think about, first part
      6060. -
    6060. More thinking
      6061. -
    6061. Still thinking
      6062. -
    6062. What does centering (subtracting the mean values) mean mathematically?
      6063. -
    6063. Further Manipulations
      6064. -
    6064. Wrapping it up
      6065. -
    6065. Linear Regression code, Intercept handling first
      6066. -
    6066. Code Examples
      6067. -
    6067. Taking out the mean
      6068. -
    6068. Friday September 9
      6069. -
    6069. Linking the regression analysis with a statistical interpretation
      6070. -
    6070. Assumptions made
      6071. -
    6071. Expectation value and variance
      6072. -
    6072. Expectation value and variance for \( \boldsymbol{\beta} \)
      6073. -
    6073. Deriving OLS from a probability distribution
      6074. -
    6074. Independent and Identically Distrubuted (iid)
      6075. -
    6075. Maximum Likelihood Estimation (MLE)
      6076. -
    6076. A new Cost Function
      6077. -
    6077. More basic Statistics and Bayes' theorem
      6078. -
    6078. Marginal Probability
      6079. -
    6079. Conditional Probability
      6080. -
    6080. Bayes' Theorem
      6081. -
    6081. Interpretations of Bayes' Theorem
      6082. -
    6082. Example of Usage of Bayes' theorem
      6083. -
    6083. Doing it correctly
      6084. -
    6084. Bayes' Theorem and Ridge and Lasso Regression
      6085. -
    6085. Test Function for what happens with OLS, Ridge and Lasso
      6086. -
    6086. Invoking Bayes' theorem
      6087. -
    6087. Ridge and Bayes
      6088. -
    6088. Lasso and Bayes
      6089. -
    6089. Exercise 1: mean values and variances in linear regression
      6090. -
    6090. Exercise 2: Adding Ridge and Lasso Regression
      6091. +
    6091. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6092. +
    6092. Linear Regression and the SVD
      6093. +
    6093. What does it mean?
      6094. +
    6094. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6095. +
    6095. Code for SVD and Inversion of Matrices
      6096. +
    6096. Inverse of Rectangular Matrix
      6097. +
    6097. Ridge and LASSO Regression
      6098. +
    6098. From OLS to Ridge and Lasso
      6099. +
    6099. Deriving the Ridge Regression Equations
      6100. +
    6100. Note on Scikit-Learn
      6101. +
    6101. Comparison with OLS
      6102. +
    6102. SVD analysis
      6103. +
    6103. Interpreting the Ridge results
      6104. +
    6104. More interpretations
      6105. +
    6105. Deriving the Lasso Regression Equations
      6106. +
    6106. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6107. +
    6107. Ridge Regression
      6108. +
    6108. Lasso Regression
      6109. +
    6109. Yet another Example
      6110. +
    6110. The OLS case
      6111. +
    6111. The Ridge case
      6112. +
    6112. Writing the Cost Function
      6113. +
    6113. Lasso case
      6114. +
    6114. The first Case
      6115. +
    6115. Simple code for solving the above problem
      6116. +
    6116. With Lasso Regression
      6117. +
    6117. Another Example, now with a polynomial fit
      6118. +
    6118. To think about, first part
      6119. +
    6119. More thinking
      6120. +
    6120. Still thinking
      6121. +
    6121. What does centering (subtracting the mean values) mean mathematically?
      6122. +
    6122. Further Manipulations
      6123. +
    6123. Wrapping it up
      6124. +
    6124. Linear Regression code, Intercept handling first
      6125. +
    6125. Code Examples
      6126. +
    6126. Taking out the mean
      6127. +
    6127. Friday September 9
      6128. +
    6128. Linking the regression analysis with a statistical interpretation
      6129. +
    6129. Assumptions made
      6130. +
    6130. Expectation value and variance
      6131. +
    6131. Expectation value and variance for \( \boldsymbol{\beta} \)
      6132. +
    6132. Deriving OLS from a probability distribution
      6133. +
    6133. Independent and Identically Distrubuted (iid)
      6134. +
    6134. Maximum Likelihood Estimation (MLE)
      6135. +
    6135. A new Cost Function
      6136. +
    6136. More basic Statistics and Bayes' theorem
      6137. +
    6137. Marginal Probability
      6138. +
    6138. Conditional Probability
      6139. +
    6139. Bayes' Theorem
      6140. +
    6140. Interpretations of Bayes' Theorem
      6141. +
    6141. Example of Usage of Bayes' theorem
      6142. +
    6142. Doing it correctly
      6143. +
    6143. Bayes' Theorem and Ridge and Lasso Regression
      6144. +
    6144. Test Function for what happens with OLS, Ridge and Lasso
      6145. +
    6145. Invoking Bayes' theorem
      6146. +
    6146. Ridge and Bayes
      6147. +
    6147. Lasso and Bayes
      6148. +
    6148. Exercise 1: mean values and variances in linear regression
      6149. +
    6149. Exercise 2: Adding Ridge and Lasso Regression
      6150. @@ -297,18 +295,20 @@ MathJax.Hub.Config({

       

       

       

      -

      Conditional Probability

      +

      Bayes' Theorem

      -

      The conditional probability, if \( p(Y) > 0 \), is

      -
      -
      - +

      If we combine the conditional probability with the marginal probability and the standard product rule, we have

      $$ -p(X\vert Y)= \frac{p(X,Y)}{p(Y)}=\frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}. +p(X\vert Y)= \frac{p(X,Y)}{p(Y)}, $$ -
      -
      +

      which we can rewrite as

      + +$$ +p(X\vert Y)= \frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}=\frac{p(Y\vert X)p(X)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}, +$$ + +

      which is Bayes' theorem. It allows us to evaluate the uncertainty in in \( X \) after we have observed \( Y \). We can easily interchange \( X \) with \( Y \).

      @@ -333,7 +333,6 @@ $$

    6150. 57
    6151. 58
    6152. 59
      6153. -
    6153. 60
    6154. »
      6155. diff --git a/doc/pub/week36/html/._week36-bs051.html b/doc/pub/week36/html/._week36-bs051.html index 3317c8a93..993271ae5 100644 --- a/doc/pub/week36/html/._week36-bs051.html +++ b/doc/pub/week36/html/._week36-bs051.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    6155. Plans for week 36
      6156. -
    6156. Thursday September 8
      6157. -
    6157. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6158. -
    6158. Linear Regression and the SVD
      6159. -
    6159. What does it mean?
      6160. -
    6160. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6161. -
    6161. Code for SVD and Inversion of Matrices
      6162. -
    6162. Inverse of Rectangular Matrix
      6163. -
    6163. Ridge and LASSO Regression
      6164. -
    6164. From OLS to Ridge and Lasso
      6165. -
    6165. Deriving the Ridge Regression Equations
      6166. -
    6166. Note on Scikit-Learn
      6167. -
    6167. Comparison with OLS
      6168. -
    6168. SVD analysis
      6169. -
    6169. Interpreting the Ridge results
      6170. -
    6170. More interpretations
      6171. -
    6171. Deriving the Lasso Regression Equations
      6172. -
    6172. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6173. -
    6173. Ridge Regression
      6174. -
    6174. Lasso Regression
      6175. -
    6175. Yet another Example
      6176. -
    6176. The OLS case
      6177. -
    6177. The Ridge case
      6178. -
    6178. Writing the Cost Function
      6179. -
    6179. Lasso case
      6180. -
    6180. The first Case
      6181. -
    6181. Simple code for solving the above problem
      6182. -
    6182. With Lasso Regression
      6183. -
    6183. Another Example, now with a polynomial fit
      6184. -
    6184. To think about, first part
      6185. -
    6185. More thinking
      6186. -
    6186. Still thinking
      6187. -
    6187. What does centering (subtracting the mean values) mean mathematically?
      6188. -
    6188. Further Manipulations
      6189. -
    6189. Wrapping it up
      6190. -
    6190. Linear Regression code, Intercept handling first
      6191. -
    6191. Code Examples
      6192. -
    6192. Taking out the mean
      6193. -
    6193. Friday September 9
      6194. -
    6194. Linking the regression analysis with a statistical interpretation
      6195. -
    6195. Assumptions made
      6196. -
    6196. Expectation value and variance
      6197. -
    6197. Expectation value and variance for \( \boldsymbol{\beta} \)
      6198. -
    6198. Deriving OLS from a probability distribution
      6199. -
    6199. Independent and Identically Distrubuted (iid)
      6200. -
    6200. Maximum Likelihood Estimation (MLE)
      6201. -
    6201. A new Cost Function
      6202. -
    6202. More basic Statistics and Bayes' theorem
      6203. -
    6203. Marginal Probability
      6204. -
    6204. Conditional Probability
      6205. -
    6205. Bayes' Theorem
      6206. -
    6206. Interpretations of Bayes' Theorem
      6207. -
    6207. Example of Usage of Bayes' theorem
      6208. -
    6208. Doing it correctly
      6209. -
    6209. Bayes' Theorem and Ridge and Lasso Regression
      6210. -
    6210. Test Function for what happens with OLS, Ridge and Lasso
      6211. -
    6211. Invoking Bayes' theorem
      6212. -
    6212. Ridge and Bayes
      6213. -
    6213. Lasso and Bayes
      6214. -
    6214. Exercise 1: mean values and variances in linear regression
      6215. -
    6215. Exercise 2: Adding Ridge and Lasso Regression
      6216. +
    6216. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6217. +
    6217. Linear Regression and the SVD
      6218. +
    6218. What does it mean?
      6219. +
    6219. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6220. +
    6220. Code for SVD and Inversion of Matrices
      6221. +
    6221. Inverse of Rectangular Matrix
      6222. +
    6222. Ridge and LASSO Regression
      6223. +
    6223. From OLS to Ridge and Lasso
      6224. +
    6224. Deriving the Ridge Regression Equations
      6225. +
    6225. Note on Scikit-Learn
      6226. +
    6226. Comparison with OLS
      6227. +
    6227. SVD analysis
      6228. +
    6228. Interpreting the Ridge results
      6229. +
    6229. More interpretations
      6230. +
    6230. Deriving the Lasso Regression Equations
      6231. +
    6231. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6232. +
    6232. Ridge Regression
      6233. +
    6233. Lasso Regression
      6234. +
    6234. Yet another Example
      6235. +
    6235. The OLS case
      6236. +
    6236. The Ridge case
      6237. +
    6237. Writing the Cost Function
      6238. +
    6238. Lasso case
      6239. +
    6239. The first Case
      6240. +
    6240. Simple code for solving the above problem
      6241. +
    6241. With Lasso Regression
      6242. +
    6242. Another Example, now with a polynomial fit
      6243. +
    6243. To think about, first part
      6244. +
    6244. More thinking
      6245. +
    6245. Still thinking
      6246. +
    6246. What does centering (subtracting the mean values) mean mathematically?
      6247. +
    6247. Further Manipulations
      6248. +
    6248. Wrapping it up
      6249. +
    6249. Linear Regression code, Intercept handling first
      6250. +
    6250. Code Examples
      6251. +
    6251. Taking out the mean
      6252. +
    6252. Friday September 9
      6253. +
    6253. Linking the regression analysis with a statistical interpretation
      6254. +
    6254. Assumptions made
      6255. +
    6255. Expectation value and variance
      6256. +
    6256. Expectation value and variance for \( \boldsymbol{\beta} \)
      6257. +
    6257. Deriving OLS from a probability distribution
      6258. +
    6258. Independent and Identically Distrubuted (iid)
      6259. +
    6259. Maximum Likelihood Estimation (MLE)
      6260. +
    6260. A new Cost Function
      6261. +
    6261. More basic Statistics and Bayes' theorem
      6262. +
    6262. Marginal Probability
      6263. +
    6263. Conditional Probability
      6264. +
    6264. Bayes' Theorem
      6265. +
    6265. Interpretations of Bayes' Theorem
      6266. +
    6266. Example of Usage of Bayes' theorem
      6267. +
    6267. Doing it correctly
      6268. +
    6268. Bayes' Theorem and Ridge and Lasso Regression
      6269. +
    6269. Test Function for what happens with OLS, Ridge and Lasso
      6270. +
    6270. Invoking Bayes' theorem
      6271. +
    6271. Ridge and Bayes
      6272. +
    6272. Lasso and Bayes
      6273. +
    6273. Exercise 1: mean values and variances in linear regression
      6274. +
    6274. Exercise 2: Adding Ridge and Lasso Regression
      6275. @@ -297,20 +295,17 @@ MathJax.Hub.Config({

       

       

       

      -

      Bayes' Theorem

      +

      Interpretations of Bayes' Theorem

      -

      If we combine the conditional probability with the marginal probability and the standard product rule, we have

      -$$ -p(X\vert Y)= \frac{p(X,Y)}{p(Y)}, -$$ +

      The quantity \( p(Y\vert X) \) on the right-hand side of the theorem is +evaluated for the observed data \( Y \) and can be viewed as a function of +the parameter space represented by \( X \). This function is not +necesseraly normalized and is normally called the likelihood function. +

      -

      which we can rewrite as

      +

      The function \( p(X) \) on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution.

      -$$ -p(X\vert Y)= \frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}=\frac{p(Y\vert X)p(X)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}, -$$ - -

      which is Bayes' theorem. It allows us to evaluate the uncertainty in in \( X \) after we have observed \( Y \). We can easily interchange \( X \) with \( Y \).

      +

      Let us try to illustrate Bayes' theorem through an example.

      @@ -334,7 +329,6 @@ $$

    6275. 57
    6276. 58
    6277. 59
      6278. -
    6278. 60
    6279. »
      6280. diff --git a/doc/pub/week36/html/._week36-bs052.html b/doc/pub/week36/html/._week36-bs052.html index 0eae500c9..64f292ff0 100644 --- a/doc/pub/week36/html/._week36-bs052.html +++ b/doc/pub/week36/html/._week36-bs052.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    6280. Plans for week 36
      6281. -
    6281. Thursday September 8
      6282. -
    6282. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6283. -
    6283. Linear Regression and the SVD
      6284. -
    6284. What does it mean?
      6285. -
    6285. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6286. -
    6286. Code for SVD and Inversion of Matrices
      6287. -
    6287. Inverse of Rectangular Matrix
      6288. -
    6288. Ridge and LASSO Regression
      6289. -
    6289. From OLS to Ridge and Lasso
      6290. -
    6290. Deriving the Ridge Regression Equations
      6291. -
    6291. Note on Scikit-Learn
      6292. -
    6292. Comparison with OLS
      6293. -
    6293. SVD analysis
      6294. -
    6294. Interpreting the Ridge results
      6295. -
    6295. More interpretations
      6296. -
    6296. Deriving the Lasso Regression Equations
      6297. -
    6297. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6298. -
    6298. Ridge Regression
      6299. -
    6299. Lasso Regression
      6300. -
    6300. Yet another Example
      6301. -
    6301. The OLS case
      6302. -
    6302. The Ridge case
      6303. -
    6303. Writing the Cost Function
      6304. -
    6304. Lasso case
      6305. -
    6305. The first Case
      6306. -
    6306. Simple code for solving the above problem
      6307. -
    6307. With Lasso Regression
      6308. -
    6308. Another Example, now with a polynomial fit
      6309. -
    6309. To think about, first part
      6310. -
    6310. More thinking
      6311. -
    6311. Still thinking
      6312. -
    6312. What does centering (subtracting the mean values) mean mathematically?
      6313. -
    6313. Further Manipulations
      6314. -
    6314. Wrapping it up
      6315. -
    6315. Linear Regression code, Intercept handling first
      6316. -
    6316. Code Examples
      6317. -
    6317. Taking out the mean
      6318. -
    6318. Friday September 9
      6319. -
    6319. Linking the regression analysis with a statistical interpretation
      6320. -
    6320. Assumptions made
      6321. -
    6321. Expectation value and variance
      6322. -
    6322. Expectation value and variance for \( \boldsymbol{\beta} \)
      6323. -
    6323. Deriving OLS from a probability distribution
      6324. -
    6324. Independent and Identically Distrubuted (iid)
      6325. -
    6325. Maximum Likelihood Estimation (MLE)
      6326. -
    6326. A new Cost Function
      6327. -
    6327. More basic Statistics and Bayes' theorem
      6328. -
    6328. Marginal Probability
      6329. -
    6329. Conditional Probability
      6330. -
    6330. Bayes' Theorem
      6331. -
    6331. Interpretations of Bayes' Theorem
      6332. -
    6332. Example of Usage of Bayes' theorem
      6333. -
    6333. Doing it correctly
      6334. -
    6334. Bayes' Theorem and Ridge and Lasso Regression
      6335. -
    6335. Test Function for what happens with OLS, Ridge and Lasso
      6336. -
    6336. Invoking Bayes' theorem
      6337. -
    6337. Ridge and Bayes
      6338. -
    6338. Lasso and Bayes
      6339. -
    6339. Exercise 1: mean values and variances in linear regression
      6340. -
    6340. Exercise 2: Adding Ridge and Lasso Regression
      6341. +
    6341. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6342. +
    6342. Linear Regression and the SVD
      6343. +
    6343. What does it mean?
      6344. +
    6344. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6345. +
    6345. Code for SVD and Inversion of Matrices
      6346. +
    6346. Inverse of Rectangular Matrix
      6347. +
    6347. Ridge and LASSO Regression
      6348. +
    6348. From OLS to Ridge and Lasso
      6349. +
    6349. Deriving the Ridge Regression Equations
      6350. +
    6350. Note on Scikit-Learn
      6351. +
    6351. Comparison with OLS
      6352. +
    6352. SVD analysis
      6353. +
    6353. Interpreting the Ridge results
      6354. +
    6354. More interpretations
      6355. +
    6355. Deriving the Lasso Regression Equations
      6356. +
    6356. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6357. +
    6357. Ridge Regression
      6358. +
    6358. Lasso Regression
      6359. +
    6359. Yet another Example
      6360. +
    6360. The OLS case
      6361. +
    6361. The Ridge case
      6362. +
    6362. Writing the Cost Function
      6363. +
    6363. Lasso case
      6364. +
    6364. The first Case
      6365. +
    6365. Simple code for solving the above problem
      6366. +
    6366. With Lasso Regression
      6367. +
    6367. Another Example, now with a polynomial fit
      6368. +
    6368. To think about, first part
      6369. +
    6369. More thinking
      6370. +
    6370. Still thinking
      6371. +
    6371. What does centering (subtracting the mean values) mean mathematically?
      6372. +
    6372. Further Manipulations
      6373. +
    6373. Wrapping it up
      6374. +
    6374. Linear Regression code, Intercept handling first
      6375. +
    6375. Code Examples
      6376. +
    6376. Taking out the mean
      6377. +
    6377. Friday September 9
      6378. +
    6378. Linking the regression analysis with a statistical interpretation
      6379. +
    6379. Assumptions made
      6380. +
    6380. Expectation value and variance
      6381. +
    6381. Expectation value and variance for \( \boldsymbol{\beta} \)
      6382. +
    6382. Deriving OLS from a probability distribution
      6383. +
    6383. Independent and Identically Distrubuted (iid)
      6384. +
    6384. Maximum Likelihood Estimation (MLE)
      6385. +
    6385. A new Cost Function
      6386. +
    6386. More basic Statistics and Bayes' theorem
      6387. +
    6387. Marginal Probability
      6388. +
    6388. Conditional Probability
      6389. +
    6389. Bayes' Theorem
      6390. +
    6390. Interpretations of Bayes' Theorem
      6391. +
    6391. Example of Usage of Bayes' theorem
      6392. +
    6392. Doing it correctly
      6393. +
    6393. Bayes' Theorem and Ridge and Lasso Regression
      6394. +
    6394. Test Function for what happens with OLS, Ridge and Lasso
      6395. +
    6395. Invoking Bayes' theorem
      6396. +
    6396. Ridge and Bayes
      6397. +
    6397. Lasso and Bayes
      6398. +
    6398. Exercise 1: mean values and variances in linear regression
      6399. +
    6399. Exercise 2: Adding Ridge and Lasso Regression
      6400. @@ -297,17 +295,27 @@ MathJax.Hub.Config({

       

       

       

      -

      Interpretations of Bayes' Theorem

      +

      Example of Usage of Bayes' theorem

      -

      The quantity \( p(Y\vert X) \) on the right-hand side of the theorem is -evaluated for the observed data \( Y \) and can be viewed as a function of -the parameter space represented by \( X \). This function is not -necesseraly normalized and is normally called the likelihood function. +

      Let us suppose that you are undergoing a series of mammography scans in +order to rule out possible breast cancer cases. We define the +sensitivity for a positive event by the variable \( X \). It takes binary +values with \( X=1 \) representing a positive event and \( X=0 \) being a +negative event. We reserve \( Y \) as a classification parameter for +either a negative or a positive breast cancer confirmation. (Short note on wordings: positive here means having breast cancer, although none of us would consider this being a positive thing).

      -

      The function \( p(X) \) on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution.

      +

      We let \( Y=1 \) represent the the case of having breast cancer and \( Y=0 \) as not.

      -

      Let us try to illustrate Bayes' theorem through an example.

      +

      Let us assume that if you have breast cancer, the test will be positive with a probability of \( 0.8 \), that is we have

      + +$$ +p(X=1\vert Y=1) =0.8. +$$ + +

      This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of \( 80\% \) for having cancer. +It is however not correct, as the following Bayesian analysis shows. +

      @@ -330,7 +338,6 @@ necesseraly normalized and is normally called the likelihood function.

    6400. 57
    6401. 58
    6402. 59
      6403. -
    6403. 60
    6404. »
      6405. diff --git a/doc/pub/week36/html/._week36-bs053.html b/doc/pub/week36/html/._week36-bs053.html index 1ee31c0ff..242731eb8 100644 --- a/doc/pub/week36/html/._week36-bs053.html +++ b/doc/pub/week36/html/._week36-bs053.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    6405. Plans for week 36
      6406. -
    6406. Thursday September 8
      6407. -
    6407. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6408. -
    6408. Linear Regression and the SVD
      6409. -
    6409. What does it mean?
      6410. -
    6410. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6411. -
    6411. Code for SVD and Inversion of Matrices
      6412. -
    6412. Inverse of Rectangular Matrix
      6413. -
    6413. Ridge and LASSO Regression
      6414. -
    6414. From OLS to Ridge and Lasso
      6415. -
    6415. Deriving the Ridge Regression Equations
      6416. -
    6416. Note on Scikit-Learn
      6417. -
    6417. Comparison with OLS
      6418. -
    6418. SVD analysis
      6419. -
    6419. Interpreting the Ridge results
      6420. -
    6420. More interpretations
      6421. -
    6421. Deriving the Lasso Regression Equations
      6422. -
    6422. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6423. -
    6423. Ridge Regression
      6424. -
    6424. Lasso Regression
      6425. -
    6425. Yet another Example
      6426. -
    6426. The OLS case
      6427. -
    6427. The Ridge case
      6428. -
    6428. Writing the Cost Function
      6429. -
    6429. Lasso case
      6430. -
    6430. The first Case
      6431. -
    6431. Simple code for solving the above problem
      6432. -
    6432. With Lasso Regression
      6433. -
    6433. Another Example, now with a polynomial fit
      6434. -
    6434. To think about, first part
      6435. -
    6435. More thinking
      6436. -
    6436. Still thinking
      6437. -
    6437. What does centering (subtracting the mean values) mean mathematically?
      6438. -
    6438. Further Manipulations
      6439. -
    6439. Wrapping it up
      6440. -
    6440. Linear Regression code, Intercept handling first
      6441. -
    6441. Code Examples
      6442. -
    6442. Taking out the mean
      6443. -
    6443. Friday September 9
      6444. -
    6444. Linking the regression analysis with a statistical interpretation
      6445. -
    6445. Assumptions made
      6446. -
    6446. Expectation value and variance
      6447. -
    6447. Expectation value and variance for \( \boldsymbol{\beta} \)
      6448. -
    6448. Deriving OLS from a probability distribution
      6449. -
    6449. Independent and Identically Distrubuted (iid)
      6450. -
    6450. Maximum Likelihood Estimation (MLE)
      6451. -
    6451. A new Cost Function
      6452. -
    6452. More basic Statistics and Bayes' theorem
      6453. -
    6453. Marginal Probability
      6454. -
    6454. Conditional Probability
      6455. -
    6455. Bayes' Theorem
      6456. -
    6456. Interpretations of Bayes' Theorem
      6457. -
    6457. Example of Usage of Bayes' theorem
      6458. -
    6458. Doing it correctly
      6459. -
    6459. Bayes' Theorem and Ridge and Lasso Regression
      6460. -
    6460. Test Function for what happens with OLS, Ridge and Lasso
      6461. -
    6461. Invoking Bayes' theorem
      6462. -
    6462. Ridge and Bayes
      6463. -
    6463. Lasso and Bayes
      6464. -
    6464. Exercise 1: mean values and variances in linear regression
      6465. -
    6465. Exercise 2: Adding Ridge and Lasso Regression
      6466. +
    6466. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6467. +
    6467. Linear Regression and the SVD
      6468. +
    6468. What does it mean?
      6469. +
    6469. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6470. +
    6470. Code for SVD and Inversion of Matrices
      6471. +
    6471. Inverse of Rectangular Matrix
      6472. +
    6472. Ridge and LASSO Regression
      6473. +
    6473. From OLS to Ridge and Lasso
      6474. +
    6474. Deriving the Ridge Regression Equations
      6475. +
    6475. Note on Scikit-Learn
      6476. +
    6476. Comparison with OLS
      6477. +
    6477. SVD analysis
      6478. +
    6478. Interpreting the Ridge results
      6479. +
    6479. More interpretations
      6480. +
    6480. Deriving the Lasso Regression Equations
      6481. +
    6481. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6482. +
    6482. Ridge Regression
      6483. +
    6483. Lasso Regression
      6484. +
    6484. Yet another Example
      6485. +
    6485. The OLS case
      6486. +
    6486. The Ridge case
      6487. +
    6487. Writing the Cost Function
      6488. +
    6488. Lasso case
      6489. +
    6489. The first Case
      6490. +
    6490. Simple code for solving the above problem
      6491. +
    6491. With Lasso Regression
      6492. +
    6492. Another Example, now with a polynomial fit
      6493. +
    6493. To think about, first part
      6494. +
    6494. More thinking
      6495. +
    6495. Still thinking
      6496. +
    6496. What does centering (subtracting the mean values) mean mathematically?
      6497. +
    6497. Further Manipulations
      6498. +
    6498. Wrapping it up
      6499. +
    6499. Linear Regression code, Intercept handling first
      6500. +
    6500. Code Examples
      6501. +
    6501. Taking out the mean
      6502. +
    6502. Friday September 9
      6503. +
    6503. Linking the regression analysis with a statistical interpretation
      6504. +
    6504. Assumptions made
      6505. +
    6505. Expectation value and variance
      6506. +
    6506. Expectation value and variance for \( \boldsymbol{\beta} \)
      6507. +
    6507. Deriving OLS from a probability distribution
      6508. +
    6508. Independent and Identically Distrubuted (iid)
      6509. +
    6509. Maximum Likelihood Estimation (MLE)
      6510. +
    6510. A new Cost Function
      6511. +
    6511. More basic Statistics and Bayes' theorem
      6512. +
    6512. Marginal Probability
      6513. +
    6513. Conditional Probability
      6514. +
    6514. Bayes' Theorem
      6515. +
    6515. Interpretations of Bayes' Theorem
      6516. +
    6516. Example of Usage of Bayes' theorem
      6517. +
    6517. Doing it correctly
      6518. +
    6518. Bayes' Theorem and Ridge and Lasso Regression
      6519. +
    6519. Test Function for what happens with OLS, Ridge and Lasso
      6520. +
    6520. Invoking Bayes' theorem
      6521. +
    6521. Ridge and Bayes
      6522. +
    6522. Lasso and Bayes
      6523. +
    6523. Exercise 1: mean values and variances in linear regression
      6524. +
    6524. Exercise 2: Adding Ridge and Lasso Regression
      6525. @@ -297,27 +295,28 @@ MathJax.Hub.Config({

       

       

       

      -

      Example of Usage of Bayes' theorem

      +

      Doing it correctly

      -

      Let us suppose that you are undergoing a series of mammography scans in -order to rule out possible breast cancer cases. We define the -sensitivity for a positive event by the variable \( X \). It takes binary -values with \( X=1 \) representing a positive event and \( X=0 \) being a -negative event. We reserve \( Y \) as a classification parameter for -either a negative or a positive breast cancer confirmation. (Short note on wordings: positive here means having breast cancer, although none of us would consider this being a positive thing). +

      If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number. +Let us assume that the prior probability in the population as a whole is

      -

      We let \( Y=1 \) represent the the case of having breast cancer and \( Y=0 \) as not.

      - -

      Let us assume that if you have breast cancer, the test will be positive with a probability of \( 0.8 \), that is we have

      - $$ -p(X=1\vert Y=1) =0.8. +p(Y=1) =0.004. $$ -

      This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of \( 80\% \) for having cancer. -It is however not correct, as the following Bayesian analysis shows. -

      +

      We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have

      +$$ +p(X=1\vert Y=0) =0.1. +$$ + +

      Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute

      + +$$ +p(Y=1\vert X=1)=\frac{p(X=1\vert Y=1)p(Y=1)}{p(X=1\vert Y=1)p(Y=1)+p(X=1\vert Y=0)p(Y=0)}=\frac{0.8\times 0.004}{0.8\times 0.004+0.1\times 0.996}=0.031. +$$ + +

      That is, in case of a positive test, there is only a \( 3\% \) chance of having breast cancer!

      @@ -339,7 +338,6 @@ It is however not correct, as the following Bayesian analysis shows.

    6525. 57
    6526. 58
    6527. 59
      6528. -
    6528. 60
    6529. »
      6530. diff --git a/doc/pub/week36/html/._week36-bs054.html b/doc/pub/week36/html/._week36-bs054.html index 6d76121de..9fb68e006 100644 --- a/doc/pub/week36/html/._week36-bs054.html +++ b/doc/pub/week36/html/._week36-bs054.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    6530. Plans for week 36
      6531. -
    6531. Thursday September 8
      6532. -
    6532. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6533. -
    6533. Linear Regression and the SVD
      6534. -
    6534. What does it mean?
      6535. -
    6535. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6536. -
    6536. Code for SVD and Inversion of Matrices
      6537. -
    6537. Inverse of Rectangular Matrix
      6538. -
    6538. Ridge and LASSO Regression
      6539. -
    6539. From OLS to Ridge and Lasso
      6540. -
    6540. Deriving the Ridge Regression Equations
      6541. -
    6541. Note on Scikit-Learn
      6542. -
    6542. Comparison with OLS
      6543. -
    6543. SVD analysis
      6544. -
    6544. Interpreting the Ridge results
      6545. -
    6545. More interpretations
      6546. -
    6546. Deriving the Lasso Regression Equations
      6547. -
    6547. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6548. -
    6548. Ridge Regression
      6549. -
    6549. Lasso Regression
      6550. -
    6550. Yet another Example
      6551. -
    6551. The OLS case
      6552. -
    6552. The Ridge case
      6553. -
    6553. Writing the Cost Function
      6554. -
    6554. Lasso case
      6555. -
    6555. The first Case
      6556. -
    6556. Simple code for solving the above problem
      6557. -
    6557. With Lasso Regression
      6558. -
    6558. Another Example, now with a polynomial fit
      6559. -
    6559. To think about, first part
      6560. -
    6560. More thinking
      6561. -
    6561. Still thinking
      6562. -
    6562. What does centering (subtracting the mean values) mean mathematically?
      6563. -
    6563. Further Manipulations
      6564. -
    6564. Wrapping it up
      6565. -
    6565. Linear Regression code, Intercept handling first
      6566. -
    6566. Code Examples
      6567. -
    6567. Taking out the mean
      6568. -
    6568. Friday September 9
      6569. -
    6569. Linking the regression analysis with a statistical interpretation
      6570. -
    6570. Assumptions made
      6571. -
    6571. Expectation value and variance
      6572. -
    6572. Expectation value and variance for \( \boldsymbol{\beta} \)
      6573. -
    6573. Deriving OLS from a probability distribution
      6574. -
    6574. Independent and Identically Distrubuted (iid)
      6575. -
    6575. Maximum Likelihood Estimation (MLE)
      6576. -
    6576. A new Cost Function
      6577. -
    6577. More basic Statistics and Bayes' theorem
      6578. -
    6578. Marginal Probability
      6579. -
    6579. Conditional Probability
      6580. -
    6580. Bayes' Theorem
      6581. -
    6581. Interpretations of Bayes' Theorem
      6582. -
    6582. Example of Usage of Bayes' theorem
      6583. -
    6583. Doing it correctly
      6584. -
    6584. Bayes' Theorem and Ridge and Lasso Regression
      6585. -
    6585. Test Function for what happens with OLS, Ridge and Lasso
      6586. -
    6586. Invoking Bayes' theorem
      6587. -
    6587. Ridge and Bayes
      6588. -
    6588. Lasso and Bayes
      6589. -
    6589. Exercise 1: mean values and variances in linear regression
      6590. -
    6590. Exercise 2: Adding Ridge and Lasso Regression
      6591. +
    6591. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6592. +
    6592. Linear Regression and the SVD
      6593. +
    6593. What does it mean?
      6594. +
    6594. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6595. +
    6595. Code for SVD and Inversion of Matrices
      6596. +
    6596. Inverse of Rectangular Matrix
      6597. +
    6597. Ridge and LASSO Regression
      6598. +
    6598. From OLS to Ridge and Lasso
      6599. +
    6599. Deriving the Ridge Regression Equations
      6600. +
    6600. Note on Scikit-Learn
      6601. +
    6601. Comparison with OLS
      6602. +
    6602. SVD analysis
      6603. +
    6603. Interpreting the Ridge results
      6604. +
    6604. More interpretations
      6605. +
    6605. Deriving the Lasso Regression Equations
      6606. +
    6606. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6607. +
    6607. Ridge Regression
      6608. +
    6608. Lasso Regression
      6609. +
    6609. Yet another Example
      6610. +
    6610. The OLS case
      6611. +
    6611. The Ridge case
      6612. +
    6612. Writing the Cost Function
      6613. +
    6613. Lasso case
      6614. +
    6614. The first Case
      6615. +
    6615. Simple code for solving the above problem
      6616. +
    6616. With Lasso Regression
      6617. +
    6617. Another Example, now with a polynomial fit
      6618. +
    6618. To think about, first part
      6619. +
    6619. More thinking
      6620. +
    6620. Still thinking
      6621. +
    6621. What does centering (subtracting the mean values) mean mathematically?
      6622. +
    6622. Further Manipulations
      6623. +
    6623. Wrapping it up
      6624. +
    6624. Linear Regression code, Intercept handling first
      6625. +
    6625. Code Examples
      6626. +
    6626. Taking out the mean
      6627. +
    6627. Friday September 9
      6628. +
    6628. Linking the regression analysis with a statistical interpretation
      6629. +
    6629. Assumptions made
      6630. +
    6630. Expectation value and variance
      6631. +
    6631. Expectation value and variance for \( \boldsymbol{\beta} \)
      6632. +
    6632. Deriving OLS from a probability distribution
      6633. +
    6633. Independent and Identically Distrubuted (iid)
      6634. +
    6634. Maximum Likelihood Estimation (MLE)
      6635. +
    6635. A new Cost Function
      6636. +
    6636. More basic Statistics and Bayes' theorem
      6637. +
    6637. Marginal Probability
      6638. +
    6638. Conditional Probability
      6639. +
    6639. Bayes' Theorem
      6640. +
    6640. Interpretations of Bayes' Theorem
      6641. +
    6641. Example of Usage of Bayes' theorem
      6642. +
    6642. Doing it correctly
      6643. +
    6643. Bayes' Theorem and Ridge and Lasso Regression
      6644. +
    6644. Test Function for what happens with OLS, Ridge and Lasso
      6645. +
    6645. Invoking Bayes' theorem
      6646. +
    6646. Ridge and Bayes
      6647. +
    6647. Lasso and Bayes
      6648. +
    6648. Exercise 1: mean values and variances in linear regression
      6649. +
    6649. Exercise 2: Adding Ridge and Lasso Regression
      6650. @@ -297,28 +295,15 @@ MathJax.Hub.Config({

       

       

       

      -

      Doing it correctly

      +

      Bayes' Theorem and Ridge and Lasso Regression

      -

      If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number. -Let us assume that the prior probability in the population as a whole is +

      Hitherto we have discussed Ridge and Lasso regression in terms of a +linear analysis. This may to many of you feel rather technical and +perhaps not that intuitive. The question is whether we can develop a +more intuitive way of understanding what Ridge and Lasso express.

      -$$ -p(Y=1) =0.004. -$$ - -

      We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have

      -$$ -p(X=1\vert Y=0) =0.1. -$$ - -

      Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute

      - -$$ -p(Y=1\vert X=1)=\frac{p(X=1\vert Y=1)p(Y=1)}{p(X=1\vert Y=1)p(Y=1)+p(X=1\vert Y=0)p(Y=0)}=\frac{0.8\times 0.004}{0.8\times 0.004+0.1\times 0.996}=0.031. -$$ - -

      That is, in case of a positive test, there is only a \( 3\% \) chance of having breast cancer!

      +

      Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit.

      @@ -339,7 +324,6 @@ $$

    6650. 57
    6651. 58
    6652. 59
      6653. -
    6653. 60
    6654. »
      6655. diff --git a/doc/pub/week36/html/._week36-bs055.html b/doc/pub/week36/html/._week36-bs055.html index 89b466920..5073e52ea 100644 --- a/doc/pub/week36/html/._week36-bs055.html +++ b/doc/pub/week36/html/._week36-bs055.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    6655. Plans for week 36
      6656. -
    6656. Thursday September 8
      6657. -
    6657. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6658. -
    6658. Linear Regression and the SVD
      6659. -
    6659. What does it mean?
      6660. -
    6660. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6661. -
    6661. Code for SVD and Inversion of Matrices
      6662. -
    6662. Inverse of Rectangular Matrix
      6663. -
    6663. Ridge and LASSO Regression
      6664. -
    6664. From OLS to Ridge and Lasso
      6665. -
    6665. Deriving the Ridge Regression Equations
      6666. -
    6666. Note on Scikit-Learn
      6667. -
    6667. Comparison with OLS
      6668. -
    6668. SVD analysis
      6669. -
    6669. Interpreting the Ridge results
      6670. -
    6670. More interpretations
      6671. -
    6671. Deriving the Lasso Regression Equations
      6672. -
    6672. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6673. -
    6673. Ridge Regression
      6674. -
    6674. Lasso Regression
      6675. -
    6675. Yet another Example
      6676. -
    6676. The OLS case
      6677. -
    6677. The Ridge case
      6678. -
    6678. Writing the Cost Function
      6679. -
    6679. Lasso case
      6680. -
    6680. The first Case
      6681. -
    6681. Simple code for solving the above problem
      6682. -
    6682. With Lasso Regression
      6683. -
    6683. Another Example, now with a polynomial fit
      6684. -
    6684. To think about, first part
      6685. -
    6685. More thinking
      6686. -
    6686. Still thinking
      6687. -
    6687. What does centering (subtracting the mean values) mean mathematically?
      6688. -
    6688. Further Manipulations
      6689. -
    6689. Wrapping it up
      6690. -
    6690. Linear Regression code, Intercept handling first
      6691. -
    6691. Code Examples
      6692. -
    6692. Taking out the mean
      6693. -
    6693. Friday September 9
      6694. -
    6694. Linking the regression analysis with a statistical interpretation
      6695. -
    6695. Assumptions made
      6696. -
    6696. Expectation value and variance
      6697. -
    6697. Expectation value and variance for \( \boldsymbol{\beta} \)
      6698. -
    6698. Deriving OLS from a probability distribution
      6699. -
    6699. Independent and Identically Distrubuted (iid)
      6700. -
    6700. Maximum Likelihood Estimation (MLE)
      6701. -
    6701. A new Cost Function
      6702. -
    6702. More basic Statistics and Bayes' theorem
      6703. -
    6703. Marginal Probability
      6704. -
    6704. Conditional Probability
      6705. -
    6705. Bayes' Theorem
      6706. -
    6706. Interpretations of Bayes' Theorem
      6707. -
    6707. Example of Usage of Bayes' theorem
      6708. -
    6708. Doing it correctly
      6709. -
    6709. Bayes' Theorem and Ridge and Lasso Regression
      6710. -
    6710. Test Function for what happens with OLS, Ridge and Lasso
      6711. -
    6711. Invoking Bayes' theorem
      6712. -
    6712. Ridge and Bayes
      6713. -
    6713. Lasso and Bayes
      6714. -
    6714. Exercise 1: mean values and variances in linear regression
      6715. -
    6715. Exercise 2: Adding Ridge and Lasso Regression
      6716. +
    6716. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6717. +
    6717. Linear Regression and the SVD
      6718. +
    6718. What does it mean?
      6719. +
    6719. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6720. +
    6720. Code for SVD and Inversion of Matrices
      6721. +
    6721. Inverse of Rectangular Matrix
      6722. +
    6722. Ridge and LASSO Regression
      6723. +
    6723. From OLS to Ridge and Lasso
      6724. +
    6724. Deriving the Ridge Regression Equations
      6725. +
    6725. Note on Scikit-Learn
      6726. +
    6726. Comparison with OLS
      6727. +
    6727. SVD analysis
      6728. +
    6728. Interpreting the Ridge results
      6729. +
    6729. More interpretations
      6730. +
    6730. Deriving the Lasso Regression Equations
      6731. +
    6731. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6732. +
    6732. Ridge Regression
      6733. +
    6733. Lasso Regression
      6734. +
    6734. Yet another Example
      6735. +
    6735. The OLS case
      6736. +
    6736. The Ridge case
      6737. +
    6737. Writing the Cost Function
      6738. +
    6738. Lasso case
      6739. +
    6739. The first Case
      6740. +
    6740. Simple code for solving the above problem
      6741. +
    6741. With Lasso Regression
      6742. +
    6742. Another Example, now with a polynomial fit
      6743. +
    6743. To think about, first part
      6744. +
    6744. More thinking
      6745. +
    6745. Still thinking
      6746. +
    6746. What does centering (subtracting the mean values) mean mathematically?
      6747. +
    6747. Further Manipulations
      6748. +
    6748. Wrapping it up
      6749. +
    6749. Linear Regression code, Intercept handling first
      6750. +
    6750. Code Examples
      6751. +
    6751. Taking out the mean
      6752. +
    6752. Friday September 9
      6753. +
    6753. Linking the regression analysis with a statistical interpretation
      6754. +
    6754. Assumptions made
      6755. +
    6755. Expectation value and variance
      6756. +
    6756. Expectation value and variance for \( \boldsymbol{\beta} \)
      6757. +
    6757. Deriving OLS from a probability distribution
      6758. +
    6758. Independent and Identically Distrubuted (iid)
      6759. +
    6759. Maximum Likelihood Estimation (MLE)
      6760. +
    6760. A new Cost Function
      6761. +
    6761. More basic Statistics and Bayes' theorem
      6762. +
    6762. Marginal Probability
      6763. +
    6763. Conditional Probability
      6764. +
    6764. Bayes' Theorem
      6765. +
    6765. Interpretations of Bayes' Theorem
      6766. +
    6766. Example of Usage of Bayes' theorem
      6767. +
    6767. Doing it correctly
      6768. +
    6768. Bayes' Theorem and Ridge and Lasso Regression
      6769. +
    6769. Test Function for what happens with OLS, Ridge and Lasso
      6770. +
    6770. Invoking Bayes' theorem
      6771. +
    6771. Ridge and Bayes
      6772. +
    6772. Lasso and Bayes
      6773. +
    6773. Exercise 1: mean values and variances in linear regression
      6774. +
    6774. Exercise 2: Adding Ridge and Lasso Regression
      6775. @@ -297,15 +295,103 @@ MathJax.Hub.Config({

       

       

       

      -

      Bayes' Theorem and Ridge and Lasso Regression

      +

      Test Function for what happens with OLS, Ridge and Lasso

      -

      Hitherto we have discussed Ridge and Lasso regression in terms of a -linear analysis. This may to many of you feel rather technical and -perhaps not that intuitive. The question is whether we can develop a -more intuitive way of understanding what Ridge and Lasso express. +

      We will play around with a study of the values for the optimal +parameters \( \boldsymbol{\beta} \) using OLS, Ridge and Lasso regression. For +OLS, you will notice as function of the noise and polynomial degree, +that the parameters \( \beta \) will fluctuate from order to order in the +polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS.

      -

      Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit.

      +

      For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one.

      + + + +
      +
      +
      +
      +
      + 
      import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+from sklearn import linear_model
+
+def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+# Make data set.
+n = 10000
+x = np.random.rand(n)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.randn(n)
+
+Maxpolydegree = 5
+X = np.zeros((len(x),Maxpolydegree))
+X[:,0] = 1.0
+
+for polydegree in range(1, Maxpolydegree):
+    for degree in range(polydegree):
+        X[:,degree] = x**(degree)
+
+
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+# matrix inversion to find beta
+OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train
+print(OLSbeta)
+ypredictOLS = X_test @ OLSbeta
+print("Test MSE OLS")
+print(MSE(y_test,ypredictOLS))
+# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn
+# Decide which values of lambda to use
+nlambdas = 4
+MSERidgePredict = np.zeros(nlambdas)
+MSELassoPredict = np.zeros(nlambdas)
+lambdas = np.logspace(-3, 1, nlambdas)
+for i in range(nlambdas):
+    lmb = lambdas[i]
+    # Make the fit using Ridge and Lasso
+    RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
+    RegRidge.fit(X_train,y_train)
+    RegLasso = linear_model.Lasso(lmb,fit_intercept=False)
+    RegLasso.fit(X_train,y_train)
+    # and then make the prediction
+    ypredictRidge = RegRidge.predict(X_test)
+    ypredictLasso = RegLasso.predict(X_test)
+    # Compute the MSE and print it
+    MSERidgePredict[i] = MSE(y_test,ypredictRidge)
+    MSELassoPredict[i] = MSE(y_test,ypredictLasso)
+    print(lmb,RegRidge.coef_)
+    print(lmb,RegLasso.coef_)
+# Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSERidgePredict, 'b', label = 'MSE Ridge Test')
+plt.plot(np.log10(lambdas), MSELassoPredict, 'r', label = 'MSE Lasso Test')
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      + +

      How can we understand this?

      @@ -325,7 +411,6 @@ more intuitive way of understanding what Ridge and Lasso express.

    6775. 57
    6776. 58
    6777. 59
      6778. -
    6778. 60
    6779. »
      6780. diff --git a/doc/pub/week36/html/._week36-bs056.html b/doc/pub/week36/html/._week36-bs056.html index 3670e8727..d675aa3bc 100644 --- a/doc/pub/week36/html/._week36-bs056.html +++ b/doc/pub/week36/html/._week36-bs056.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    6780. Plans for week 36
      6781. -
    6781. Thursday September 8
      6782. -
    6782. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6783. -
    6783. Linear Regression and the SVD
      6784. -
    6784. What does it mean?
      6785. -
    6785. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6786. -
    6786. Code for SVD and Inversion of Matrices
      6787. -
    6787. Inverse of Rectangular Matrix
      6788. -
    6788. Ridge and LASSO Regression
      6789. -
    6789. From OLS to Ridge and Lasso
      6790. -
    6790. Deriving the Ridge Regression Equations
      6791. -
    6791. Note on Scikit-Learn
      6792. -
    6792. Comparison with OLS
      6793. -
    6793. SVD analysis
      6794. -
    6794. Interpreting the Ridge results
      6795. -
    6795. More interpretations
      6796. -
    6796. Deriving the Lasso Regression Equations
      6797. -
    6797. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6798. -
    6798. Ridge Regression
      6799. -
    6799. Lasso Regression
      6800. -
    6800. Yet another Example
      6801. -
    6801. The OLS case
      6802. -
    6802. The Ridge case
      6803. -
    6803. Writing the Cost Function
      6804. -
    6804. Lasso case
      6805. -
    6805. The first Case
      6806. -
    6806. Simple code for solving the above problem
      6807. -
    6807. With Lasso Regression
      6808. -
    6808. Another Example, now with a polynomial fit
      6809. -
    6809. To think about, first part
      6810. -
    6810. More thinking
      6811. -
    6811. Still thinking
      6812. -
    6812. What does centering (subtracting the mean values) mean mathematically?
      6813. -
    6813. Further Manipulations
      6814. -
    6814. Wrapping it up
      6815. -
    6815. Linear Regression code, Intercept handling first
      6816. -
    6816. Code Examples
      6817. -
    6817. Taking out the mean
      6818. -
    6818. Friday September 9
      6819. -
    6819. Linking the regression analysis with a statistical interpretation
      6820. -
    6820. Assumptions made
      6821. -
    6821. Expectation value and variance
      6822. -
    6822. Expectation value and variance for \( \boldsymbol{\beta} \)
      6823. -
    6823. Deriving OLS from a probability distribution
      6824. -
    6824. Independent and Identically Distrubuted (iid)
      6825. -
    6825. Maximum Likelihood Estimation (MLE)
      6826. -
    6826. A new Cost Function
      6827. -
    6827. More basic Statistics and Bayes' theorem
      6828. -
    6828. Marginal Probability
      6829. -
    6829. Conditional Probability
      6830. -
    6830. Bayes' Theorem
      6831. -
    6831. Interpretations of Bayes' Theorem
      6832. -
    6832. Example of Usage of Bayes' theorem
      6833. -
    6833. Doing it correctly
      6834. -
    6834. Bayes' Theorem and Ridge and Lasso Regression
      6835. -
    6835. Test Function for what happens with OLS, Ridge and Lasso
      6836. -
    6836. Invoking Bayes' theorem
      6837. -
    6837. Ridge and Bayes
      6838. -
    6838. Lasso and Bayes
      6839. -
    6839. Exercise 1: mean values and variances in linear regression
      6840. -
    6840. Exercise 2: Adding Ridge and Lasso Regression
      6841. +
    6841. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6842. +
    6842. Linear Regression and the SVD
      6843. +
    6843. What does it mean?
      6844. +
    6844. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6845. +
    6845. Code for SVD and Inversion of Matrices
      6846. +
    6846. Inverse of Rectangular Matrix
      6847. +
    6847. Ridge and LASSO Regression
      6848. +
    6848. From OLS to Ridge and Lasso
      6849. +
    6849. Deriving the Ridge Regression Equations
      6850. +
    6850. Note on Scikit-Learn
      6851. +
    6851. Comparison with OLS
      6852. +
    6852. SVD analysis
      6853. +
    6853. Interpreting the Ridge results
      6854. +
    6854. More interpretations
      6855. +
    6855. Deriving the Lasso Regression Equations
      6856. +
    6856. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6857. +
    6857. Ridge Regression
      6858. +
    6858. Lasso Regression
      6859. +
    6859. Yet another Example
      6860. +
    6860. The OLS case
      6861. +
    6861. The Ridge case
      6862. +
    6862. Writing the Cost Function
      6863. +
    6863. Lasso case
      6864. +
    6864. The first Case
      6865. +
    6865. Simple code for solving the above problem
      6866. +
    6866. With Lasso Regression
      6867. +
    6867. Another Example, now with a polynomial fit
      6868. +
    6868. To think about, first part
      6869. +
    6869. More thinking
      6870. +
    6870. Still thinking
      6871. +
    6871. What does centering (subtracting the mean values) mean mathematically?
      6872. +
    6872. Further Manipulations
      6873. +
    6873. Wrapping it up
      6874. +
    6874. Linear Regression code, Intercept handling first
      6875. +
    6875. Code Examples
      6876. +
    6876. Taking out the mean
      6877. +
    6877. Friday September 9
      6878. +
    6878. Linking the regression analysis with a statistical interpretation
      6879. +
    6879. Assumptions made
      6880. +
    6880. Expectation value and variance
      6881. +
    6881. Expectation value and variance for \( \boldsymbol{\beta} \)
      6882. +
    6882. Deriving OLS from a probability distribution
      6883. +
    6883. Independent and Identically Distrubuted (iid)
      6884. +
    6884. Maximum Likelihood Estimation (MLE)
      6885. +
    6885. A new Cost Function
      6886. +
    6886. More basic Statistics and Bayes' theorem
      6887. +
    6887. Marginal Probability
      6888. +
    6888. Conditional Probability
      6889. +
    6889. Bayes' Theorem
      6890. +
    6890. Interpretations of Bayes' Theorem
      6891. +
    6891. Example of Usage of Bayes' theorem
      6892. +
    6892. Doing it correctly
      6893. +
    6893. Bayes' Theorem and Ridge and Lasso Regression
      6894. +
    6894. Test Function for what happens with OLS, Ridge and Lasso
      6895. +
    6895. Invoking Bayes' theorem
      6896. +
    6896. Ridge and Bayes
      6897. +
    6897. Lasso and Bayes
      6898. +
    6898. Exercise 1: mean values and variances in linear regression
      6899. +
    6899. Exercise 2: Adding Ridge and Lasso Regression
      6900. @@ -297,103 +295,32 @@ MathJax.Hub.Config({

       

       

       

      -

      Test Function for what happens with OLS, Ridge and Lasso

      +

      Invoking Bayes' theorem

      -

      We will play around with a study of the values for the optimal -parameters \( \boldsymbol{\beta} \) using OLS, Ridge and Lasso regression. For -OLS, you will notice as function of the noise and polynomial degree, -that the parameters \( \beta \) will fluctuate from order to order in the -polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS. -

      +

      Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression.

      -

      For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one.

      +

      For ordinary least squares we postulated that the maximum likelihood for the doamin of events \( \boldsymbol{D} \) (one-dimensional case)

      +$$ +\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})], +$$ +

      is given by

      +$$ +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +$$ - -
      -
      -
      -
      -
      - 
      import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-from sklearn import linear_model
+
      In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \( \boldsymbol{\beta} \) given a domain of events \( \boldsymbol{D} \)?  That is, how can we define the posterior probability 
      
 
-def R2(y_data, y_model):
-    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-def MSE(y_data,y_model):
-    n = np.size(y_model)
-    return np.sum((y_data-y_model)**2)/n
+$$
+p(\boldsymbol{\beta}\vert\boldsymbol{D}).
+$$
 
-# Make data set.
-n = 10000
-x = np.random.rand(n)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.randn(n)
+
      Bayes' theorem comes to our rescue here since (omitting the normalization constant)
      
+$$
+p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}).
+$$
 
-Maxpolydegree = 5
-X = np.zeros((len(x),Maxpolydegree))
-X[:,0] = 1.0
-
-for polydegree in range(1, Maxpolydegree):
-    for degree in range(polydegree):
-        X[:,degree] = x**(degree)
-
-
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-
-# matrix inversion to find beta
-OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train
-print(OLSbeta)
-ypredictOLS = X_test @ OLSbeta
-print("Test MSE OLS")
-print(MSE(y_test,ypredictOLS))
-# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn
-# Decide which values of lambda to use
-nlambdas = 4
-MSERidgePredict = np.zeros(nlambdas)
-MSELassoPredict = np.zeros(nlambdas)
-lambdas = np.logspace(-3, 1, nlambdas)
-for i in range(nlambdas):
-    lmb = lambdas[i]
-    # Make the fit using Ridge and Lasso
-    RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
-    RegRidge.fit(X_train,y_train)
-    RegLasso = linear_model.Lasso(lmb,fit_intercept=False)
-    RegLasso.fit(X_train,y_train)
-    # and then make the prediction
-    ypredictRidge = RegRidge.predict(X_test)
-    ypredictLasso = RegLasso.predict(X_test)
-    # Compute the MSE and print it
-    MSERidgePredict[i] = MSE(y_test,ypredictRidge)
-    MSELassoPredict[i] = MSE(y_test,ypredictLasso)
-    print(lmb,RegRidge.coef_)
-    print(lmb,RegLasso.coef_)
-# Now plot the results
-plt.figure()
-plt.plot(np.log10(lambdas), MSERidgePredict, 'b', label = 'MSE Ridge Test')
-plt.plot(np.log10(lambdas), MSELassoPredict, 'r', label = 'MSE Lasso Test')
-plt.xlabel('log10(lambda)')
-plt.ylabel('MSE')
-plt.legend()
-plt.show()
-
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      -
      - -

      How can we understand this?

      +

      We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one for the prior \( p(\boldsymbol{\beta} \)!

      @@ -412,7 +339,6 @@ plt.show()

    6900. 57
    6901. 58
    6902. 59
      6903. -
    6903. 60
    6904. »
      6905. diff --git a/doc/pub/week36/html/._week36-bs057.html b/doc/pub/week36/html/._week36-bs057.html index 57f956961..aa93d7987 100644 --- a/doc/pub/week36/html/._week36-bs057.html +++ b/doc/pub/week36/html/._week36-bs057.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    6905. Plans for week 36
      6906. -
    6906. Thursday September 8
      6907. -
    6907. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6908. -
    6908. Linear Regression and the SVD
      6909. -
    6909. What does it mean?
      6910. -
    6910. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6911. -
    6911. Code for SVD and Inversion of Matrices
      6912. -
    6912. Inverse of Rectangular Matrix
      6913. -
    6913. Ridge and LASSO Regression
      6914. -
    6914. From OLS to Ridge and Lasso
      6915. -
    6915. Deriving the Ridge Regression Equations
      6916. -
    6916. Note on Scikit-Learn
      6917. -
    6917. Comparison with OLS
      6918. -
    6918. SVD analysis
      6919. -
    6919. Interpreting the Ridge results
      6920. -
    6920. More interpretations
      6921. -
    6921. Deriving the Lasso Regression Equations
      6922. -
    6922. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6923. -
    6923. Ridge Regression
      6924. -
    6924. Lasso Regression
      6925. -
    6925. Yet another Example
      6926. -
    6926. The OLS case
      6927. -
    6927. The Ridge case
      6928. -
    6928. Writing the Cost Function
      6929. -
    6929. Lasso case
      6930. -
    6930. The first Case
      6931. -
    6931. Simple code for solving the above problem
      6932. -
    6932. With Lasso Regression
      6933. -
    6933. Another Example, now with a polynomial fit
      6934. -
    6934. To think about, first part
      6935. -
    6935. More thinking
      6936. -
    6936. Still thinking
      6937. -
    6937. What does centering (subtracting the mean values) mean mathematically?
      6938. -
    6938. Further Manipulations
      6939. -
    6939. Wrapping it up
      6940. -
    6940. Linear Regression code, Intercept handling first
      6941. -
    6941. Code Examples
      6942. -
    6942. Taking out the mean
      6943. -
    6943. Friday September 9
      6944. -
    6944. Linking the regression analysis with a statistical interpretation
      6945. -
    6945. Assumptions made
      6946. -
    6946. Expectation value and variance
      6947. -
    6947. Expectation value and variance for \( \boldsymbol{\beta} \)
      6948. -
    6948. Deriving OLS from a probability distribution
      6949. -
    6949. Independent and Identically Distrubuted (iid)
      6950. -
    6950. Maximum Likelihood Estimation (MLE)
      6951. -
    6951. A new Cost Function
      6952. -
    6952. More basic Statistics and Bayes' theorem
      6953. -
    6953. Marginal Probability
      6954. -
    6954. Conditional Probability
      6955. -
    6955. Bayes' Theorem
      6956. -
    6956. Interpretations of Bayes' Theorem
      6957. -
    6957. Example of Usage of Bayes' theorem
      6958. -
    6958. Doing it correctly
      6959. -
    6959. Bayes' Theorem and Ridge and Lasso Regression
      6960. -
    6960. Test Function for what happens with OLS, Ridge and Lasso
      6961. -
    6961. Invoking Bayes' theorem
      6962. -
    6962. Ridge and Bayes
      6963. -
    6963. Lasso and Bayes
      6964. -
    6964. Exercise 1: mean values and variances in linear regression
      6965. -
    6965. Exercise 2: Adding Ridge and Lasso Regression
      6966. +
    6966. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      6967. +
    6967. Linear Regression and the SVD
      6968. +
    6968. What does it mean?
      6969. +
    6969. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      6970. +
    6970. Code for SVD and Inversion of Matrices
      6971. +
    6971. Inverse of Rectangular Matrix
      6972. +
    6972. Ridge and LASSO Regression
      6973. +
    6973. From OLS to Ridge and Lasso
      6974. +
    6974. Deriving the Ridge Regression Equations
      6975. +
    6975. Note on Scikit-Learn
      6976. +
    6976. Comparison with OLS
      6977. +
    6977. SVD analysis
      6978. +
    6978. Interpreting the Ridge results
      6979. +
    6979. More interpretations
      6980. +
    6980. Deriving the Lasso Regression Equations
      6981. +
    6981. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      6982. +
    6982. Ridge Regression
      6983. +
    6983. Lasso Regression
      6984. +
    6984. Yet another Example
      6985. +
    6985. The OLS case
      6986. +
    6986. The Ridge case
      6987. +
    6987. Writing the Cost Function
      6988. +
    6988. Lasso case
      6989. +
    6989. The first Case
      6990. +
    6990. Simple code for solving the above problem
      6991. +
    6991. With Lasso Regression
      6992. +
    6992. Another Example, now with a polynomial fit
      6993. +
    6993. To think about, first part
      6994. +
    6994. More thinking
      6995. +
    6995. Still thinking
      6996. +
    6996. What does centering (subtracting the mean values) mean mathematically?
      6997. +
    6997. Further Manipulations
      6998. +
    6998. Wrapping it up
      6999. +
    6999. Linear Regression code, Intercept handling first
      7000. +
    7000. Code Examples
      7001. +
    7001. Taking out the mean
      7002. +
    7002. Friday September 9
      7003. +
    7003. Linking the regression analysis with a statistical interpretation
      7004. +
    7004. Assumptions made
      7005. +
    7005. Expectation value and variance
      7006. +
    7006. Expectation value and variance for \( \boldsymbol{\beta} \)
      7007. +
    7007. Deriving OLS from a probability distribution
      7008. +
    7008. Independent and Identically Distrubuted (iid)
      7009. +
    7009. Maximum Likelihood Estimation (MLE)
      7010. +
    7010. A new Cost Function
      7011. +
    7011. More basic Statistics and Bayes' theorem
      7012. +
    7012. Marginal Probability
      7013. +
    7013. Conditional Probability
      7014. +
    7014. Bayes' Theorem
      7015. +
    7015. Interpretations of Bayes' Theorem
      7016. +
    7016. Example of Usage of Bayes' theorem
      7017. +
    7017. Doing it correctly
      7018. +
    7018. Bayes' Theorem and Ridge and Lasso Regression
      7019. +
    7019. Test Function for what happens with OLS, Ridge and Lasso
      7020. +
    7020. Invoking Bayes' theorem
      7021. +
    7021. Ridge and Bayes
      7022. +
    7022. Lasso and Bayes
      7023. +
    7023. Exercise 1: mean values and variances in linear regression
      7024. +
    7024. Exercise 2: Adding Ridge and Lasso Regression
      7025. @@ -297,32 +295,41 @@ MathJax.Hub.Config({

       

       

       

      -

      Invoking Bayes' theorem

      +

      Ridge and Bayes

      -

      Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression.

      +

      With the posterior probability defined by a likelihood which we have +already modeled and an unknown prior, we are now ready to make +additional models for the prior. +

      -

      For ordinary least squares we postulated that the maximum likelihood for the doamin of events \( \boldsymbol{D} \) (one-dimensional case)

      -$$ -\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})], -$$ - -

      is given by

      -$$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. -$$ - -

      In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \( \boldsymbol{\beta} \) given a domain of events \( \boldsymbol{D} \)? That is, how can we define the posterior probability

      +

      We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is

      $$ -p(\boldsymbol{\beta}\vert\boldsymbol{D}). +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. $$ -

      Bayes' theorem comes to our rescue here since (omitting the normalization constant)

      +

      Our posterior probability becomes then (omitting the normalization factor which is just a constant)

      $$ -p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}). +p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. $$ -

      We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one for the prior \( p(\boldsymbol{\beta} \)!

      +

      We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we +did for OLS, this is most conveniently done by taking the negative +logarithm of the posterior probability. Doing so and leaving out the +constants terms that do not depend on \( \beta \), we have +

      + +$$ +C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ + +

      and replacing \( 1/2\tau^2 \) with \( \lambda \) we have

      + +$$ +C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ + +

      which is our Ridge cost function! Nice, isn't it?

      @@ -340,7 +347,6 @@ $$

    7025. 57
    7026. 58
    7027. 59
      7028. -
    7028. 60
    7029. »
      7030. diff --git a/doc/pub/week36/html/._week36-bs058.html b/doc/pub/week36/html/._week36-bs058.html index 0e365e002..75d994099 100644 --- a/doc/pub/week36/html/._week36-bs058.html +++ b/doc/pub/week36/html/._week36-bs058.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    7030. Plans for week 36
      7031. -
    7031. Thursday September 8
      7032. -
    7032. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      7033. -
    7033. Linear Regression and the SVD
      7034. -
    7034. What does it mean?
      7035. -
    7035. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      7036. -
    7036. Code for SVD and Inversion of Matrices
      7037. -
    7037. Inverse of Rectangular Matrix
      7038. -
    7038. Ridge and LASSO Regression
      7039. -
    7039. From OLS to Ridge and Lasso
      7040. -
    7040. Deriving the Ridge Regression Equations
      7041. -
    7041. Note on Scikit-Learn
      7042. -
    7042. Comparison with OLS
      7043. -
    7043. SVD analysis
      7044. -
    7044. Interpreting the Ridge results
      7045. -
    7045. More interpretations
      7046. -
    7046. Deriving the Lasso Regression Equations
      7047. -
    7047. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      7048. -
    7048. Ridge Regression
      7049. -
    7049. Lasso Regression
      7050. -
    7050. Yet another Example
      7051. -
    7051. The OLS case
      7052. -
    7052. The Ridge case
      7053. -
    7053. Writing the Cost Function
      7054. -
    7054. Lasso case
      7055. -
    7055. The first Case
      7056. -
    7056. Simple code for solving the above problem
      7057. -
    7057. With Lasso Regression
      7058. -
    7058. Another Example, now with a polynomial fit
      7059. -
    7059. To think about, first part
      7060. -
    7060. More thinking
      7061. -
    7061. Still thinking
      7062. -
    7062. What does centering (subtracting the mean values) mean mathematically?
      7063. -
    7063. Further Manipulations
      7064. -
    7064. Wrapping it up
      7065. -
    7065. Linear Regression code, Intercept handling first
      7066. -
    7066. Code Examples
      7067. -
    7067. Taking out the mean
      7068. -
    7068. Friday September 9
      7069. -
    7069. Linking the regression analysis with a statistical interpretation
      7070. -
    7070. Assumptions made
      7071. -
    7071. Expectation value and variance
      7072. -
    7072. Expectation value and variance for \( \boldsymbol{\beta} \)
      7073. -
    7073. Deriving OLS from a probability distribution
      7074. -
    7074. Independent and Identically Distrubuted (iid)
      7075. -
    7075. Maximum Likelihood Estimation (MLE)
      7076. -
    7076. A new Cost Function
      7077. -
    7077. More basic Statistics and Bayes' theorem
      7078. -
    7078. Marginal Probability
      7079. -
    7079. Conditional Probability
      7080. -
    7080. Bayes' Theorem
      7081. -
    7081. Interpretations of Bayes' Theorem
      7082. -
    7082. Example of Usage of Bayes' theorem
      7083. -
    7083. Doing it correctly
      7084. -
    7084. Bayes' Theorem and Ridge and Lasso Regression
      7085. -
    7085. Test Function for what happens with OLS, Ridge and Lasso
      7086. -
    7086. Invoking Bayes' theorem
      7087. -
    7087. Ridge and Bayes
      7088. -
    7088. Lasso and Bayes
      7089. -
    7089. Exercise 1: mean values and variances in linear regression
      7090. -
    7090. Exercise 2: Adding Ridge and Lasso Regression
      7091. +
    7091. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      7092. +
    7092. Linear Regression and the SVD
      7093. +
    7093. What does it mean?
      7094. +
    7094. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      7095. +
    7095. Code for SVD and Inversion of Matrices
      7096. +
    7096. Inverse of Rectangular Matrix
      7097. +
    7097. Ridge and LASSO Regression
      7098. +
    7098. From OLS to Ridge and Lasso
      7099. +
    7099. Deriving the Ridge Regression Equations
      7100. +
    7100. Note on Scikit-Learn
      7101. +
    7101. Comparison with OLS
      7102. +
    7102. SVD analysis
      7103. +
    7103. Interpreting the Ridge results
      7104. +
    7104. More interpretations
      7105. +
    7105. Deriving the Lasso Regression Equations
      7106. +
    7106. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      7107. +
    7107. Ridge Regression
      7108. +
    7108. Lasso Regression
      7109. +
    7109. Yet another Example
      7110. +
    7110. The OLS case
      7111. +
    7111. The Ridge case
      7112. +
    7112. Writing the Cost Function
      7113. +
    7113. Lasso case
      7114. +
    7114. The first Case
      7115. +
    7115. Simple code for solving the above problem
      7116. +
    7116. With Lasso Regression
      7117. +
    7117. Another Example, now with a polynomial fit
      7118. +
    7118. To think about, first part
      7119. +
    7119. More thinking
      7120. +
    7120. Still thinking
      7121. +
    7121. What does centering (subtracting the mean values) mean mathematically?
      7122. +
    7122. Further Manipulations
      7123. +
    7123. Wrapping it up
      7124. +
    7124. Linear Regression code, Intercept handling first
      7125. +
    7125. Code Examples
      7126. +
    7126. Taking out the mean
      7127. +
    7127. Friday September 9
      7128. +
    7128. Linking the regression analysis with a statistical interpretation
      7129. +
    7129. Assumptions made
      7130. +
    7130. Expectation value and variance
      7131. +
    7131. Expectation value and variance for \( \boldsymbol{\beta} \)
      7132. +
    7132. Deriving OLS from a probability distribution
      7133. +
    7133. Independent and Identically Distrubuted (iid)
      7134. +
    7134. Maximum Likelihood Estimation (MLE)
      7135. +
    7135. A new Cost Function
      7136. +
    7136. More basic Statistics and Bayes' theorem
      7137. +
    7137. Marginal Probability
      7138. +
    7138. Conditional Probability
      7139. +
    7139. Bayes' Theorem
      7140. +
    7140. Interpretations of Bayes' Theorem
      7141. +
    7141. Example of Usage of Bayes' theorem
      7142. +
    7142. Doing it correctly
      7143. +
    7143. Bayes' Theorem and Ridge and Lasso Regression
      7144. +
    7144. Test Function for what happens with OLS, Ridge and Lasso
      7145. +
    7145. Invoking Bayes' theorem
      7146. +
    7146. Ridge and Bayes
      7147. +
    7147. Lasso and Bayes
      7148. +
    7148. Exercise 1: mean values and variances in linear regression
      7149. +
    7149. Exercise 2: Adding Ridge and Lasso Regression
      7150. @@ -297,42 +295,172 @@ MathJax.Hub.Config({

       

       

       

      -

      Ridge and Bayes

      +

      Lasso and Bayes

      -

      With the posterior probability defined by a likelihood which we have -already modeled and an unknown prior, we are now ready to make -additional models for the prior. -

      - -

      We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is

      +

      To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is

      $$ -p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. $$

      Our posterior probability becomes then (omitting the normalization factor which is just a constant)

      $$ -p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. +p(\boldsymbol{\beta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. $$ -

      We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we -did for OLS, this is most conveniently done by taking the negative -logarithm of the posterior probability. Doing so and leaving out the +

      Taking the negative +logarithm of the posterior probability and leaving out the constants terms that do not depend on \( \beta \), we have

      $$ -C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -

      and replacing \( 1/2\tau^2 \) with \( \lambda \) we have

      +

      and replacing \( 1/\tau \) with \( \lambda \) we have

      $$ -C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -

      which is our Ridge cost function! Nice, isn't it?

      +

      which is our Lasso cost function!

      + +

      Exercise 1: mean values and variances in linear regression

      + +

      This exercise deals with various mean values ad variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).

      + +

      The assumption we have made is +that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) +which describes our data +

      +$$ +\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} +$$ + +

      We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our +function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we minimized \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), with +

      +$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. +$$ + +

      The matrix \( \boldsymbol{X} \) is the so-called design matrix.

      + + +

      +a) +Show that the expectation value of \( \boldsymbol{y} \) for a given element \( i \) +

      +$$ +\begin{align*} +\mathbb{E}(y_i) & =\mathbf{X}_{i, \ast} \, \beta, +\end{align*} +$$ + +

      and that +its variance is +

      +$$ +\begin{align*} \mbox{Var}(y_i) & = \sigma^2. +\end{align*} +$$ + +

      Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \). +

      + + + + +

      +b) +With the OLS expressions for the parameters \( \boldsymbol{\beta} \) show that +

      +$$ +\mathbb{E}(\boldsymbol{\beta}) = \boldsymbol{\beta}. +$$ + + + + + +

      +c) +Show finally that the variance of \( \boldsymbol{\beta} \) is +

      +$$ +\begin{eqnarray*} +\mbox{Var}(\boldsymbol{\beta}) & = & \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}. +\end{eqnarray*} +$$ + + + + + + + +

      Exercise 2: Adding Ridge and Lasso Regression

      + +

      This exercise is a continuation of the exercises from week 35.

      + +

      We will +use the same function to generate our data set, still staying with a +simple function \( y(x) \) which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. +

      + +

      We will thus again generate our own dataset for a function \( y(x) \) where +\( x \in [0,1] \) and defined by random numbers computed with the uniform +distribution. The function \( y \) is a quadratic polynomial in \( x \) with +added stochastic noise according to the normal distribution \( \cal{N}(0,1) \). +

      + +

      The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).

      + + +
      +
      +
      +
      +
      + 
      x = np.random.rand(100)
+y = 2.0+5*x*x+0.1*np.random.randn(100)
+
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      +
      + + + +

      +a) +Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \). +

      + + + + +

      +b) +Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using Scikit-Learn to find the parameters \( \beta \). From your results calculate the variance of these parameters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)). Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie et al. and their figures 3.10 and 3.11. Scikit-Learn may not provide the variance of the parameters \( \beta \). This needs to be checked. With your own code you can however do so. +

      + + + +

      diff --git a/doc/pub/week36/html/week36-bs.html b/doc/pub/week36/html/week36-bs.html index c79950484..12be7d203 100644 --- a/doc/pub/week36/html/week36-bs.html +++ b/doc/pub/week36/html/week36-bs.html @@ -37,7 +37,6 @@ doconce format html week36.do.txt --html_style=bootstrap --pygments_html_style=d
    7150. Plans for week 36
      7151. -
    7151. Thursday September 8
      7152. -
    7152. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      7153. -
    7153. Linear Regression and the SVD
      7154. -
    7154. What does it mean?
      7155. -
    7155. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      7156. -
    7156. Code for SVD and Inversion of Matrices
      7157. -
    7157. Inverse of Rectangular Matrix
      7158. -
    7158. Ridge and LASSO Regression
      7159. -
    7159. From OLS to Ridge and Lasso
      7160. -
    7160. Deriving the Ridge Regression Equations
      7161. -
    7161. Note on Scikit-Learn
      7162. -
    7162. Comparison with OLS
      7163. -
    7163. SVD analysis
      7164. -
    7164. Interpreting the Ridge results
      7165. -
    7165. More interpretations
      7166. -
    7166. Deriving the Lasso Regression Equations
      7167. -
    7167. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      7168. -
    7168. Ridge Regression
      7169. -
    7169. Lasso Regression
      7170. -
    7170. Yet another Example
      7171. -
    7171. The OLS case
      7172. -
    7172. The Ridge case
      7173. -
    7173. Writing the Cost Function
      7174. -
    7174. Lasso case
      7175. -
    7175. The first Case
      7176. -
    7176. Simple code for solving the above problem
      7177. -
    7177. With Lasso Regression
      7178. -
    7178. Another Example, now with a polynomial fit
      7179. -
    7179. To think about, first part
      7180. -
    7180. More thinking
      7181. -
    7181. Still thinking
      7182. -
    7182. What does centering (subtracting the mean values) mean mathematically?
      7183. -
    7183. Further Manipulations
      7184. -
    7184. Wrapping it up
      7185. -
    7185. Linear Regression code, Intercept handling first
      7186. -
    7186. Code Examples
      7187. -
    7187. Taking out the mean
      7188. -
    7188. Friday September 9
      7189. -
    7189. Linking the regression analysis with a statistical interpretation
      7190. -
    7190. Assumptions made
      7191. -
    7191. Expectation value and variance
      7192. -
    7192. Expectation value and variance for \( \boldsymbol{\beta} \)
      7193. -
    7193. Deriving OLS from a probability distribution
      7194. -
    7194. Independent and Identically Distrubuted (iid)
      7195. -
    7195. Maximum Likelihood Estimation (MLE)
      7196. -
    7196. A new Cost Function
      7197. -
    7197. More basic Statistics and Bayes' theorem
      7198. -
    7198. Marginal Probability
      7199. -
    7199. Conditional Probability
      7200. -
    7200. Bayes' Theorem
      7201. -
    7201. Interpretations of Bayes' Theorem
      7202. -
    7202. Example of Usage of Bayes' theorem
      7203. -
    7203. Doing it correctly
      7204. -
    7204. Bayes' Theorem and Ridge and Lasso Regression
      7205. -
    7205. Test Function for what happens with OLS, Ridge and Lasso
      7206. -
    7206. Invoking Bayes' theorem
      7207. -
    7207. Ridge and Bayes
      7208. -
    7208. Lasso and Bayes
      7209. -
    7209. Exercise 1: mean values and variances in linear regression
      7210. -
    7210. Exercise 2: Adding Ridge and Lasso Regression
      7211. +
    7211. Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
      7212. +
    7212. Linear Regression and the SVD
      7213. +
    7213. What does it mean?
      7214. +
    7214. And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
      7215. +
    7215. Code for SVD and Inversion of Matrices
      7216. +
    7216. Inverse of Rectangular Matrix
      7217. +
    7217. Ridge and LASSO Regression
      7218. +
    7218. From OLS to Ridge and Lasso
      7219. +
    7219. Deriving the Ridge Regression Equations
      7220. +
    7220. Note on Scikit-Learn
      7221. +
    7221. Comparison with OLS
      7222. +
    7222. SVD analysis
      7223. +
    7223. Interpreting the Ridge results
      7224. +
    7224. More interpretations
      7225. +
    7225. Deriving the Lasso Regression Equations
      7226. +
    7226. Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
      7227. +
    7227. Ridge Regression
      7228. +
    7228. Lasso Regression
      7229. +
    7229. Yet another Example
      7230. +
    7230. The OLS case
      7231. +
    7231. The Ridge case
      7232. +
    7232. Writing the Cost Function
      7233. +
    7233. Lasso case
      7234. +
    7234. The first Case
      7235. +
    7235. Simple code for solving the above problem
      7236. +
    7236. With Lasso Regression
      7237. +
    7237. Another Example, now with a polynomial fit
      7238. +
    7238. To think about, first part
      7239. +
    7239. More thinking
      7240. +
    7240. Still thinking
      7241. +
    7241. What does centering (subtracting the mean values) mean mathematically?
      7242. +
    7242. Further Manipulations
      7243. +
    7243. Wrapping it up
      7244. +
    7244. Linear Regression code, Intercept handling first
      7245. +
    7245. Code Examples
      7246. +
    7246. Taking out the mean
      7247. +
    7247. Friday September 9
      7248. +
    7248. Linking the regression analysis with a statistical interpretation
      7249. +
    7249. Assumptions made
      7250. +
    7250. Expectation value and variance
      7251. +
    7251. Expectation value and variance for \( \boldsymbol{\beta} \)
      7252. +
    7252. Deriving OLS from a probability distribution
      7253. +
    7253. Independent and Identically Distrubuted (iid)
      7254. +
    7254. Maximum Likelihood Estimation (MLE)
      7255. +
    7255. A new Cost Function
      7256. +
    7256. More basic Statistics and Bayes' theorem
      7257. +
    7257. Marginal Probability
      7258. +
    7258. Conditional Probability
      7259. +
    7259. Bayes' Theorem
      7260. +
    7260. Interpretations of Bayes' Theorem
      7261. +
    7261. Example of Usage of Bayes' theorem
      7262. +
    7262. Doing it correctly
      7263. +
    7263. Bayes' Theorem and Ridge and Lasso Regression
      7264. +
    7264. Test Function for what happens with OLS, Ridge and Lasso
      7265. +
    7265. Invoking Bayes' theorem
      7266. +
    7266. Ridge and Bayes
      7267. +
    7267. Lasso and Bayes
      7268. +
    7268. Exercise 1: mean values and variances in linear regression
      7269. +
    7269. Exercise 2: Adding Ridge and Lasso Regression
      7270. @@ -315,7 +313,7 @@ MathJax.Hub.Config({
      -

      Sep 9, 2022

      +

      May 29, 2023


      @@ -340,7 +338,7 @@ MathJax.Hub.Config({
    7270. 9
    7271. 10
    7272. ...
      7273. -
    7273. 60
      7274. +
    7274. 59
    7275. »
      7276. @@ -354,7 +352,7 @@ MathJax.Hub.Config({ -->
      - © 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
      diff --git a/doc/pub/week36/html/week36-reveal.html b/doc/pub/week36/html/week36-reveal.html index b03749e38..79ad92daf 100644 --- a/doc/pub/week36/html/week36-reveal.html +++ b/doc/pub/week36/html/week36-reveal.html @@ -184,13 +184,13 @@ MathJax.Hub.Config({
      -

      Sep 9, 2022

      +

      May 29, 2023


      - © 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
      @@ -198,18 +198,8 @@ MathJax.Hub.Config({

      Plans for week 36

      Recommended Reading:

      @@ -220,10 +210,6 @@ MathJax.Hub.Config({
    -
    -

    Thursday September 8

    -
    -

    Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples

    diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html index d7c58a7ed..ac9235d19 100644 --- a/doc/pub/week36/html/week36-solarized.html +++ b/doc/pub/week36/html/week36-solarized.html @@ -64,7 +64,6 @@ div.toc p,a {
    @@ -263,14 +262,8 @@ MathJax.Hub.Config({

    Plans for week 36

    Recommended Reading:

      @@ -278,9 +271,6 @@ MathJax.Hub.Config({
    1. Bishop 1.1, 1.2, 2.1, 2.2, 2.3 and 3.1
    2. Hastie et al chapter 3
    -









    -

    Thursday September 8

    -









    Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples

    @@ -2533,7 +2523,7 @@ Our next step is to study the variance of the parameters \( \beta_1 \) and \( \b
    - © 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index 65f853f33..dd2e0cc28 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -141,7 +141,6 @@ div.toc p,a {
    @@ -340,14 +339,8 @@ MathJax.Hub.Config({

    Plans for week 36

    Recommended Reading:

      @@ -355,9 +348,6 @@ MathJax.Hub.Config({
    1. Bishop 1.1, 1.2, 2.1, 2.2, 2.3 and 3.1
    2. Hastie et al chapter 3
    -









    -

    Thursday September 8

    -









    Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples

    @@ -2610,7 +2600,7 @@ Our next step is to study the variance of the parameters \( \beta_1 \) and \( \b
    - © 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license + © 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index 9dd28a12a..c634bf9f0 100644 Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index ed649678e..6be1b27d0 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "54f68878", - "metadata": {}, + "id": "de098695", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,31 +14,31 @@ }, { "cell_type": "markdown", - "id": "b2606fb6", - "metadata": {}, + "id": "9d896464", + "metadata": { + "editable": true + }, "source": [ "# Week 36: Statistical interpretation of Linear Regression and Resampling techniques\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 9, 2022**\n", + "Date: **May 29, 2023**\n", "\n", - "Copyright 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" + "Copyright 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" ] }, { "cell_type": "markdown", - "id": "998031f2", - "metadata": {}, + "id": "2d144eda", + "metadata": { + "editable": true + }, "source": [ "## Plans for week 36\n", "\n", - "* Thursday: Summary from last week on discussion of SVD, Ridge and Lasso linear regression.\n", + "* Summary from last week on discussion of SVD, Ridge and Lasso linear regression.\n", "\n", - " * [Video of Lecture](https://youtu.be/qn_BAVhMD8U)\n", - "\n", - "* Friday: Linear Regression and links with Statistics, Resampling methods and presentation of first project\n", - "\n", - " * [Video of Lecture](https://youtu.be/_CPGg0JYH8M)\n", + "* Linear Regression and links with Statistics, Resampling methods and presentation of first project\n", "\n", "Recommended Reading:\n", "1. Lectures on Regression\n", @@ -48,24 +50,20 @@ }, { "cell_type": "markdown", - "id": "6cd06197", - "metadata": {}, - "source": [ - "## Thursday September 8" - ] - }, - { - "cell_type": "markdown", - "id": "d54cdcb7", - "metadata": {}, + "id": "de778f2b", + "metadata": { + "editable": true + }, "source": [ "## Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples" ] }, { "cell_type": "markdown", - "id": "e00e68d6", - "metadata": {}, + "id": "ca2b6500", + "metadata": { + "editable": true + }, "source": [ "## Linear Regression and the SVD\n", "\n", @@ -74,8 +72,10 @@ }, { "cell_type": "markdown", - "id": "34b87b68", - "metadata": {}, + "id": "a42c6c2d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -84,16 +84,20 @@ }, { "cell_type": "markdown", - "id": "ce084898", - "metadata": {}, + "id": "b42bcaa4", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined last week the matrix" ] }, { "cell_type": "markdown", - "id": "2976c1d5", - "metadata": {}, + "id": "e208741e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\end{bmatrix},\n", @@ -102,16 +106,20 @@ }, { "cell_type": "markdown", - "id": "6a83cdb5", - "metadata": {}, + "id": "1ca945f1", + "metadata": { + "editable": true + }, "source": [ "where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is" ] }, { "cell_type": "markdown", - "id": "a7a4fa16", - "metadata": {}, + "id": "0df614ad", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n", @@ -125,16 +133,20 @@ }, { "cell_type": "markdown", - "id": "03e6fafb", - "metadata": {}, + "id": "07ba6f53", + "metadata": { + "editable": true + }, "source": [ "meaning we can write" ] }, { "cell_type": "markdown", - "id": "7b638f4f", - "metadata": {}, + "id": "05d566fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n", @@ -143,16 +155,20 @@ }, { "cell_type": "markdown", - "id": "ec26d490", - "metadata": {}, + "id": "bef58181", + "metadata": { + "editable": true + }, "source": [ "Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] }, { "cell_type": "markdown", - "id": "89a41f1b", - "metadata": {}, + "id": "88faff60", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n", @@ -161,8 +177,10 @@ }, { "cell_type": "markdown", - "id": "1a1ec91f", - "metadata": {}, + "id": "78db1996", + "metadata": { + "editable": true + }, "source": [ "## What does it mean?\n", "\n", @@ -173,8 +191,10 @@ }, { "cell_type": "markdown", - "id": "9be07751", - "metadata": {}, + "id": "bacb305d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -183,8 +203,10 @@ }, { "cell_type": "markdown", - "id": "71861426", - "metadata": {}, + "id": "f72c3acc", + "metadata": { + "editable": true + }, "source": [ "In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n", "square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n", @@ -203,8 +225,10 @@ }, { "cell_type": "markdown", - "id": "8696c3c0", - "metadata": {}, + "id": "70d77e0a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n", @@ -213,8 +237,10 @@ }, { "cell_type": "markdown", - "id": "9cf87ecc", - "metadata": {}, + "id": "4e09c60a", + "metadata": { + "editable": true + }, "source": [ "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n", "the number of samples) are the eigenvalues of the covariance\n", @@ -226,8 +252,10 @@ }, { "cell_type": "markdown", - "id": "20032abb", - "metadata": {}, + "id": "454ed107", + "metadata": { + "editable": true + }, "source": [ "## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "\n", @@ -236,8 +264,10 @@ }, { "cell_type": "markdown", - "id": "fbcc9b3f", - "metadata": {}, + "id": "e680ab8d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n", @@ -246,16 +276,20 @@ }, { "cell_type": "markdown", - "id": "1e18e19f", - "metadata": {}, + "id": "55e8064a", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $n\\times n$, we have" ] }, { "cell_type": "markdown", - "id": "747b9d45", - "metadata": {}, + "id": "54323d93", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -264,16 +298,20 @@ }, { "cell_type": "markdown", - "id": "134925f1", - "metadata": {}, + "id": "11308aeb", + "metadata": { + "editable": true + }, "source": [ "leading to" ] }, { "cell_type": "markdown", - "id": "0050225b", - "metadata": {}, + "id": "e6329c2b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n", @@ -282,16 +320,20 @@ }, { "cell_type": "markdown", - "id": "8ec325de", - "metadata": {}, + "id": "850c985b", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem" ] }, { "cell_type": "markdown", - "id": "5059b729", - "metadata": {}, + "id": "967c3ea7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n", @@ -300,8 +342,10 @@ }, { "cell_type": "markdown", - "id": "1815abe6", - "metadata": {}, + "id": "482029c6", + "metadata": { + "editable": true + }, "source": [ "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n", "the non-zero singular values plus now a series of zeros. The column\n", @@ -315,8 +359,10 @@ }, { "cell_type": "markdown", - "id": "ade9b7ed", - "metadata": {}, + "id": "dc0004a9", + "metadata": { + "editable": true + }, "source": [ "## Code for SVD and Inversion of Matrices\n", "\n", @@ -327,8 +373,11 @@ { "cell_type": "code", "execution_count": 1, - "id": "24749dad", - "metadata": {}, + "id": "534ac439", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "Ainv = np.linlag.pinv(A)" @@ -336,40 +385,23 @@ }, { "cell_type": "markdown", - "id": "770bef7d", - "metadata": {}, + "id": "03919b93", + "metadata": { + "editable": true + }, "source": [ "Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD." ] }, { "cell_type": "code", - "execution_count": 4, - "id": "23a63ba1", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[[ 1. -1. 2.]\n", - " [ 1. 0. 1.]\n", - " [ 1. 2. -1.]\n", - " [ 1. 1. 0.]]\n", - "test U\n", - "[[-3.33066907e-16 -1.11022302e-16 3.33066907e-16]\n", - " [-1.11022302e-16 4.44089210e-16 -2.49800181e-16]\n", - " [ 3.33066907e-16 -2.49800181e-16 0.00000000e+00]]\n", - "test VT\n", - "[[ 2.22044605e-16 5.55111512e-17 -2.22044605e-16]\n", - " [ 5.55111512e-17 -2.22044605e-16 5.55111512e-17]\n", - " [-2.22044605e-16 5.55111512e-17 2.22044605e-16]]\n", - "[[1.82969604e+30 1.82969604e+30 1.82969604e+30]\n", - " [1.82969604e+30 1.82969604e+30 1.82969604e+30]\n", - " [1.82969604e+30 1.82969604e+30 1.82969604e+30]]\n" - ] - } - ], + "execution_count": 2, + "id": "f765d0a0", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import numpy as np\n", "# SVD inversion\n", @@ -391,21 +423,23 @@ " return np.matmul(V,np.matmul(invD,UT))\n", "\n", "\n", - "X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])\n", + "#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])\n", "# Non-singular square matrix\n", - "#X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])\n", + "X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])\n", "print(X)\n", "A = np.transpose(X) @ X\n", "# Brute force inversion\n", - "B = np.linalg.pinv(A) # here we could use np.linalg.pinv(A)\n", + "B = np.linalg.inv(A) # here we could use np.linalg.pinv(A)\n", "C = SVDinv(A)\n", "print(np.abs(B-C))" ] }, { "cell_type": "markdown", - "id": "7bd93f48", - "metadata": {}, + "id": "e459209e", + "metadata": { + "editable": true + }, "source": [ "## Inverse of Rectangular Matrix\n", "\n", @@ -422,8 +456,10 @@ }, { "cell_type": "markdown", - "id": "3fd0e27b", - "metadata": {}, + "id": "6c6da8a6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}_{\\mathrm{PI}}= \\boldsymbol{V}\\boldsymbol{D}_{\\mathrm{PI}}\\boldsymbol{U}^T,\n", @@ -432,33 +468,23 @@ }, { "cell_type": "markdown", - "id": "067d781b", - "metadata": {}, + "id": "efd71633", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{D}_{\\mathrm{PI}}$ can be calculated by creating a diagonal matrix from $\\boldsymbol{\\Sigma}$ where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD." ] }, { "cell_type": "code", - "execution_count": 5, - "id": "c17f9502", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[[0.3 0.4]\n", - " [0.5 0.6]\n", - " [0.7 0.8]\n", - " [0.9 1. ]]\n", - "[[-13. -6. 1. 8. ]\n", - " [ 11.5 5.5 -0.5 -6.5]]\n", - "[[0. 0. 0. 0.]\n", - " [0. 0. 0. 0.]]\n" - ] - } - ], + "execution_count": 3, + "id": "db9d7882", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import numpy as np\n", "# SVD inversion\n", @@ -487,16 +513,20 @@ }, { "cell_type": "markdown", - "id": "c9e2f968", - "metadata": {}, + "id": "7fee3c4d", + "metadata": { + "editable": true + }, "source": [ "As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by **Numpy**." ] }, { "cell_type": "markdown", - "id": "715bfc44", - "metadata": {}, + "id": "09aa560a", + "metadata": { + "editable": true + }, "source": [ "## Ridge and LASSO Regression\n", "\n", @@ -506,8 +536,10 @@ }, { "cell_type": "markdown", - "id": "a7250920", - "metadata": {}, + "id": "4b628c4f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -516,16 +548,20 @@ }, { "cell_type": "markdown", - "id": "2d4eca08", - "metadata": {}, + "id": "56abee21", + "metadata": { + "editable": true + }, "source": [ "or we can state it as" ] }, { "cell_type": "markdown", - "id": "5c47d606", - "metadata": {}, + "id": "aeef03ba", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -535,16 +571,20 @@ }, { "cell_type": "markdown", - "id": "be48173a", - "metadata": {}, + "id": "b0ded245", + "metadata": { + "editable": true + }, "source": [ "where we have used the definition of a norm-2 vector, that is" ] }, { "cell_type": "markdown", - "id": "c1cc0c52", - "metadata": {}, + "id": "832509eb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -553,8 +593,10 @@ }, { "cell_type": "markdown", - "id": "85043b85", - "metadata": {}, + "id": "146dc50a", + "metadata": { + "editable": true + }, "source": [ "## From OLS to Ridge and Lasso\n", "\n", @@ -566,8 +608,10 @@ }, { "cell_type": "markdown", - "id": "8b28ed2a", - "metadata": {}, + "id": "aec12b0f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -577,8 +621,10 @@ }, { "cell_type": "markdown", - "id": "498c8111", - "metadata": {}, + "id": "1326c8e8", + "metadata": { + "editable": true + }, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -587,8 +633,10 @@ }, { "cell_type": "markdown", - "id": "9d41fbd8", - "metadata": {}, + "id": "b68d575d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -597,16 +645,20 @@ }, { "cell_type": "markdown", - "id": "6bacb19c", - "metadata": {}, + "id": "11e6ed59", + "metadata": { + "editable": true + }, "source": [ "we have a new optimization equation" ] }, { "cell_type": "markdown", - "id": "4e386cd1", - "metadata": {}, + "id": "e6e1e17e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -616,8 +668,10 @@ }, { "cell_type": "markdown", - "id": "1bd20824", - "metadata": {}, + "id": "957ec3e8", + "metadata": { + "editable": true + }, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "\n", @@ -626,8 +680,10 @@ }, { "cell_type": "markdown", - "id": "4264b9bb", - "metadata": {}, + "id": "9aca13ad", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -636,8 +692,10 @@ }, { "cell_type": "markdown", - "id": "eec0c847", - "metadata": {}, + "id": "786b818b", + "metadata": { + "editable": true + }, "source": [ "## Deriving the Ridge Regression Equations\n", "\n", @@ -646,8 +704,10 @@ }, { "cell_type": "markdown", - "id": "e11a479f", - "metadata": {}, + "id": "923eb0db", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", @@ -656,8 +716,10 @@ }, { "cell_type": "markdown", - "id": "0f614521", - "metadata": {}, + "id": "d65e713e", + "metadata": { + "editable": true + }, "source": [ "and \n", "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", @@ -668,8 +730,10 @@ }, { "cell_type": "markdown", - "id": "e0a2c65a", - "metadata": {}, + "id": "3392d021", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -678,16 +742,20 @@ }, { "cell_type": "markdown", - "id": "9a36d518", - "metadata": {}, + "id": "beea32ad", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" ] }, { "cell_type": "markdown", - "id": "ec99d6ff", - "metadata": {}, + "id": "8c9054f2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", @@ -696,16 +764,20 @@ }, { "cell_type": "markdown", - "id": "e5102c81", - "metadata": {}, + "id": "ce0df680", + "metadata": { + "editable": true + }, "source": [ "with $t$ a finite positive number." ] }, { "cell_type": "markdown", - "id": "277c29d3", - "metadata": {}, + "id": "c85bcd8f", + "metadata": { + "editable": true + }, "source": [ "## Note on Scikit-Learn\n", "\n", @@ -714,8 +786,10 @@ }, { "cell_type": "markdown", - "id": "3ee1172e", - "metadata": {}, + "id": "3d57bfb2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -724,16 +798,20 @@ }, { "cell_type": "markdown", - "id": "5555becd", - "metadata": {}, + "id": "4be71669", + "metadata": { + "editable": true + }, "source": [ "In our codes where we compare our own codes with **Scikit-Learn**, we do thus not include the $1/n$ factor in the cost function." ] }, { "cell_type": "markdown", - "id": "bf920b7a", - "metadata": {}, + "id": "78291c69", + "metadata": { + "editable": true + }, "source": [ "## Comparison with OLS\n", "When we compare this with the ordinary least squares result we have" @@ -741,8 +819,10 @@ }, { "cell_type": "markdown", - "id": "dcc8b3a0", - "metadata": {}, + "id": "8e2ce9db", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -751,8 +831,10 @@ }, { "cell_type": "markdown", - "id": "b51d29a2", - "metadata": {}, + "id": "e5733ed1", + "metadata": { + "editable": true + }, "source": [ "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", @@ -765,8 +847,10 @@ }, { "cell_type": "markdown", - "id": "4c81e874", - "metadata": {}, + "id": "168c1b1b", + "metadata": { + "editable": true + }, "source": [ "## SVD analysis\n", "\n", @@ -776,8 +860,10 @@ }, { "cell_type": "markdown", - "id": "7e9a88c2", - "metadata": {}, + "id": "c9876061", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", @@ -786,16 +872,20 @@ }, { "cell_type": "markdown", - "id": "16c44a1f", - "metadata": {}, + "id": "b5e3df11", + "metadata": { + "editable": true + }, "source": [ "For Ridge regression this becomes" ] }, { "cell_type": "markdown", - "id": "be091747", - "metadata": {}, + "id": "d86a5a2a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", @@ -804,16 +894,20 @@ }, { "cell_type": "markdown", - "id": "e7c4958c", - "metadata": {}, + "id": "f905e4b4", + "metadata": { + "editable": true + }, "source": [ "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$." ] }, { "cell_type": "markdown", - "id": "096042d4", - "metadata": {}, + "id": "ab780b24", + "metadata": { + "editable": true + }, "source": [ "## Interpreting the Ridge results\n", "\n", @@ -822,8 +916,10 @@ }, { "cell_type": "markdown", - "id": "a6133162", - "metadata": {}, + "id": "d634de61", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", @@ -832,8 +928,10 @@ }, { "cell_type": "markdown", - "id": "de47ccce", - "metadata": {}, + "id": "670a41a0", + "metadata": { + "editable": true + }, "source": [ "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", @@ -846,8 +944,10 @@ }, { "cell_type": "markdown", - "id": "38d8e676", - "metadata": {}, + "id": "a66e2e86", + "metadata": { + "editable": true + }, "source": [ "## More interpretations\n", "\n", @@ -856,8 +956,10 @@ }, { "cell_type": "markdown", - "id": "cf226622", - "metadata": {}, + "id": "d7be9ae4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", @@ -866,16 +968,20 @@ }, { "cell_type": "markdown", - "id": "e51ab72f", - "metadata": {}, + "id": "417c1432", + "metadata": { + "editable": true + }, "source": [ "In this case the standard OLS results in" ] }, { "cell_type": "markdown", - "id": "aded2c2f", - "metadata": {}, + "id": "bf21b1eb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{n-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n", @@ -884,16 +990,20 @@ }, { "cell_type": "markdown", - "id": "d18586ec", - "metadata": {}, + "id": "e8c6c561", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "d60836ed", - "metadata": {}, + "id": "cc83e17d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", @@ -902,8 +1012,10 @@ }, { "cell_type": "markdown", - "id": "83ee48f4", - "metadata": {}, + "id": "85e44116", + "metadata": { + "editable": true + }, "source": [ "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", "the Ridge estimator converges to zero when the hyperparameter goes to\n", @@ -917,8 +1029,10 @@ }, { "cell_type": "markdown", - "id": "6379bc41", - "metadata": {}, + "id": "743ab24a", + "metadata": { + "editable": true + }, "source": [ "## Deriving the Lasso Regression Equations\n", "\n", @@ -927,8 +1041,10 @@ }, { "cell_type": "markdown", - "id": "60a42dcc", - "metadata": {}, + "id": "cf09ce78", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -937,16 +1053,20 @@ }, { "cell_type": "markdown", - "id": "d335d8fe", - "metadata": {}, + "id": "9bdf27d8", + "metadata": { + "editable": true + }, "source": [ "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" ] }, { "cell_type": "markdown", - "id": "f7988d24", - "metadata": {}, + "id": "1397b054", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n", @@ -955,16 +1075,20 @@ }, { "cell_type": "markdown", - "id": "455c0597", - "metadata": {}, + "id": "6d232041", + "metadata": { + "editable": true + }, "source": [ "we have that the derivative of the cost function is" ] }, { "cell_type": "markdown", - "id": "31b80ae8", - "metadata": {}, + "id": "7c743042", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", @@ -973,16 +1097,20 @@ }, { "cell_type": "markdown", - "id": "89cfcd2e", - "metadata": {}, + "id": "ecf32c47", + "metadata": { + "editable": true + }, "source": [ "and reordering we have" ] }, { "cell_type": "markdown", - "id": "7b9c1855", - "metadata": {}, + "id": "04d7809b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -991,16 +1119,20 @@ }, { "cell_type": "markdown", - "id": "64b3eb85", - "metadata": {}, + "id": "9ebd04d9", + "metadata": { + "editable": true + }, "source": [ "This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." ] }, { "cell_type": "markdown", - "id": "57e33e83", - "metadata": {}, + "id": "b9aa9a0a", + "metadata": { + "editable": true + }, "source": [ "## Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression\n", "\n", @@ -1012,8 +1144,10 @@ }, { "cell_type": "markdown", - "id": "9aaf5adc", - "metadata": {}, + "id": "d912d78a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2,\n", @@ -1022,16 +1156,20 @@ }, { "cell_type": "markdown", - "id": "a7e0855d", - "metadata": {}, + "id": "71180921", + "metadata": { + "editable": true + }, "source": [ "and minimizing we have that" ] }, { "cell_type": "markdown", - "id": "282ec43f", - "metadata": {}, + "id": "24877aab", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{OLS}} = y_i.\n", @@ -1040,8 +1178,10 @@ }, { "cell_type": "markdown", - "id": "9a8686a0", - "metadata": {}, + "id": "5ceac1f1", + "metadata": { + "editable": true + }, "source": [ "## Ridge Regression\n", "\n", @@ -1050,8 +1190,10 @@ }, { "cell_type": "markdown", - "id": "d0c7441a", - "metadata": {}, + "id": "a43fbf8e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n", @@ -1060,16 +1202,20 @@ }, { "cell_type": "markdown", - "id": "123a953a", - "metadata": {}, + "id": "ada51a1b", + "metadata": { + "editable": true + }, "source": [ "and minimizing we have that" ] }, { "cell_type": "markdown", - "id": "b80e95f5", - "metadata": {}, + "id": "eccc52bb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n", @@ -1078,8 +1224,10 @@ }, { "cell_type": "markdown", - "id": "1272391e", - "metadata": {}, + "id": "83b0d869", + "metadata": { + "editable": true + }, "source": [ "## Lasso Regression\n", "\n", @@ -1088,8 +1236,10 @@ }, { "cell_type": "markdown", - "id": "8a65a800", - "metadata": {}, + "id": "32c3342e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n", @@ -1098,16 +1248,20 @@ }, { "cell_type": "markdown", - "id": "129b29dc", - "metadata": {}, + "id": "63ea1201", + "metadata": { + "editable": true + }, "source": [ "and minimizing we have that" ] }, { "cell_type": "markdown", - "id": "9671fc3b", - "metadata": {}, + "id": "ae009692", + "metadata": { + "editable": true + }, "source": [ "$$\n", "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n", @@ -1116,16 +1270,20 @@ }, { "cell_type": "markdown", - "id": "645d20a1", - "metadata": {}, + "id": "afbd034f", + "metadata": { + "editable": true + }, "source": [ "which leads to" ] }, { "cell_type": "markdown", - "id": "4a940735", - "metadata": {}, + "id": "4917d37f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n", @@ -1136,16 +1294,20 @@ }, { "cell_type": "markdown", - "id": "f5d24fed", - "metadata": {}, + "id": "987f0c76", + "metadata": { + "editable": true + }, "source": [ "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$." ] }, { "cell_type": "markdown", - "id": "56fe26cc", - "metadata": {}, + "id": "97d73103", + "metadata": { + "editable": true + }, "source": [ "## Yet another Example\n", "\n", @@ -1154,8 +1316,10 @@ }, { "cell_type": "markdown", - "id": "bcdecc9d", - "metadata": {}, + "id": "1bb0bc4f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}=\\begin{bmatrix}4 \\\\ 2 \\\\3\\end{bmatrix},\n", @@ -1164,16 +1328,20 @@ }, { "cell_type": "markdown", - "id": "088b5e9c", - "metadata": {}, + "id": "60fa4c41", + "metadata": { + "editable": true + }, "source": [ "and our inputs as a $3\\times 2$ design matrix" ] }, { "cell_type": "markdown", - "id": "813decf7", - "metadata": {}, + "id": "9868a55b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 0 & 0\\end{bmatrix},\n", @@ -1182,16 +1350,20 @@ }, { "cell_type": "markdown", - "id": "d2887852", - "metadata": {}, + "id": "e04ddc75", + "metadata": { + "editable": true + }, "source": [ "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression." ] }, { "cell_type": "markdown", - "id": "9d7b7953", - "metadata": {}, + "id": "50b5d084", + "metadata": { + "editable": true + }, "source": [ "## The OLS case\n", "\n", @@ -1200,8 +1372,10 @@ }, { "cell_type": "markdown", - "id": "e837f852", - "metadata": {}, + "id": "4e7afb4b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1210,16 +1384,20 @@ }, { "cell_type": "markdown", - "id": "15d802eb", - "metadata": {}, + "id": "b02ec7ca", + "metadata": { + "editable": true + }, "source": [ "Inserting the above values we obtain that" ] }, { "cell_type": "markdown", - "id": "cd4b96b1", - "metadata": {}, + "id": "685a6243", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n", @@ -1228,16 +1406,20 @@ }, { "cell_type": "markdown", - "id": "434b97ea", - "metadata": {}, + "id": "05887870", + "metadata": { + "editable": true + }, "source": [ "The code which implements this simpler case is presented after the discussion of Ridge and Lasso." ] }, { "cell_type": "markdown", - "id": "d640a5fb", - "metadata": {}, + "id": "9b746aac", + "metadata": { + "editable": true + }, "source": [ "## The Ridge case\n", "\n", @@ -1246,8 +1428,10 @@ }, { "cell_type": "markdown", - "id": "f8a077e1", - "metadata": {}, + "id": "247a5768", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1256,16 +1440,20 @@ }, { "cell_type": "markdown", - "id": "30897a80", - "metadata": {}, + "id": "4fcd1df5", + "metadata": { + "editable": true + }, "source": [ "Inserting the above values we obtain that" ] }, { "cell_type": "markdown", - "id": "1e751907", - "metadata": {}, + "id": "438c8cca", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", @@ -1274,8 +1462,10 @@ }, { "cell_type": "markdown", - "id": "ca66337b", - "metadata": {}, + "id": "c55b0860", + "metadata": { + "editable": true + }, "source": [ "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n", "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n", @@ -1285,8 +1475,10 @@ }, { "cell_type": "markdown", - "id": "30cb3a5b", - "metadata": {}, + "id": "74cf0cde", + "metadata": { + "editable": true + }, "source": [ "## Writing the Cost Function\n", "\n", @@ -1295,8 +1487,10 @@ }, { "cell_type": "markdown", - "id": "365400dc", - "metadata": {}, + "id": "8e25d3a0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n", @@ -1305,8 +1499,10 @@ }, { "cell_type": "markdown", - "id": "f09ebaf1", - "metadata": {}, + "id": "f2dce6dc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n", @@ -1315,16 +1511,20 @@ }, { "cell_type": "markdown", - "id": "1de97d2c", - "metadata": {}, + "id": "7e466c2e", + "metadata": { + "editable": true + }, "source": [ "and taking the derivative with respect to $\\beta_0$ we get" ] }, { "cell_type": "markdown", - "id": "78632ac0", - "metadata": {}, + "id": "1794597d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0=\\frac{8}{4+\\lambda},\n", @@ -1333,16 +1533,20 @@ }, { "cell_type": "markdown", - "id": "3c88d85f", - "metadata": {}, + "id": "0c2a81c3", + "metadata": { + "editable": true + }, "source": [ "and for $\\beta_1$ we obtain" ] }, { "cell_type": "markdown", - "id": "97140392", - "metadata": {}, + "id": "6078099e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_1=\\frac{2}{1+\\lambda},\n", @@ -1351,16 +1555,20 @@ }, { "cell_type": "markdown", - "id": "7d1ea782", - "metadata": {}, + "id": "4a82e7ac", + "metadata": { + "editable": true + }, "source": [ "Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving" ] }, { "cell_type": "markdown", - "id": "9b49be00", - "metadata": {}, + "id": "5c4dea33", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\frac{8}{4+\\lambda}\\right)^2+\\left(\\frac{2}{1+\\lambda}\\right)^2=1,\n", @@ -1369,16 +1577,20 @@ }, { "cell_type": "markdown", - "id": "7bb324bc", - "metadata": {}, + "id": "cb4a101d", + "metadata": { + "editable": true + }, "source": [ "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$." ] }, { "cell_type": "markdown", - "id": "610a2a56", - "metadata": {}, + "id": "0d32288d", + "metadata": { + "editable": true + }, "source": [ "## Lasso case\n", "\n", @@ -1388,8 +1600,10 @@ }, { "cell_type": "markdown", - "id": "fc93bd4d", - "metadata": {}, + "id": "75dd3e83", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n", @@ -1398,8 +1612,10 @@ }, { "cell_type": "markdown", - "id": "7ac09787", - "metadata": {}, + "id": "47d4704a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n", @@ -1408,16 +1624,20 @@ }, { "cell_type": "markdown", - "id": "a7b2f817", - "metadata": {}, + "id": "3127a2c8", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "99632928", - "metadata": {}, + "id": "175b8660", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n", @@ -1426,8 +1646,10 @@ }, { "cell_type": "markdown", - "id": "94c7a534", - "metadata": {}, + "id": "f820cdbe", + "metadata": { + "editable": true + }, "source": [ "We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n", "1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n", @@ -1441,8 +1663,10 @@ }, { "cell_type": "markdown", - "id": "e55fc2b2", - "metadata": {}, + "id": "b004066b", + "metadata": { + "editable": true + }, "source": [ "## The first Case\n", "\n", @@ -1451,8 +1675,10 @@ }, { "cell_type": "markdown", - "id": "e2ef7286", - "metadata": {}, + "id": "4def35de", + "metadata": { + "editable": true + }, "source": [ "$$\n", "-4(4-2\\beta_0)+\\lambda=0,\n", @@ -1461,16 +1687,20 @@ }, { "cell_type": "markdown", - "id": "3a5c871e", - "metadata": {}, + "id": "7fdd8604", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "b9ca6fd3", - "metadata": {}, + "id": "a33da947", + "metadata": { + "editable": true + }, "source": [ "$$\n", "-2(2-\\beta_1)+\\lambda=0.\n", @@ -1479,16 +1709,20 @@ }, { "cell_type": "markdown", - "id": "4c69db28", - "metadata": {}, + "id": "8ad8d519", + "metadata": { + "editable": true + }, "source": [ "which yields" ] }, { "cell_type": "markdown", - "id": "c77c93cb", - "metadata": {}, + "id": "9637d8ec", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0=\\frac{16+\\lambda}{8},\n", @@ -1497,16 +1731,20 @@ }, { "cell_type": "markdown", - "id": "cfdc629c", - "metadata": {}, + "id": "a50871a8", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "07e71e01", - "metadata": {}, + "id": "2152879d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_1=\\frac{4+\\lambda}{2}.\n", @@ -1515,16 +1753,20 @@ }, { "cell_type": "markdown", - "id": "c8f65682", - "metadata": {}, + "id": "717ca01d", + "metadata": { + "editable": true + }, "source": [ "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you." ] }, { "cell_type": "markdown", - "id": "bc6233b2", - "metadata": {}, + "id": "cbedfde8", + "metadata": { + "editable": true + }, "source": [ "## Simple code for solving the above problem\n", "\n", @@ -1536,8 +1778,11 @@ { "cell_type": "code", "execution_count": 4, - "id": "d594a12f", - "metadata": {}, + "id": "cced0760", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "%matplotlib inline\n", @@ -1595,16 +1840,20 @@ }, { "cell_type": "markdown", - "id": "6bd62e0a", - "metadata": {}, + "id": "5cc9d0bb", + "metadata": { + "editable": true + }, "source": [ "We see here that we reach a plateau. What is actually happening?" ] }, { "cell_type": "markdown", - "id": "715aeaa5", - "metadata": {}, + "id": "8f26fc0f", + "metadata": { + "editable": true + }, "source": [ "## With Lasso Regression" ] @@ -1612,8 +1861,11 @@ { "cell_type": "code", "execution_count": 5, - "id": "84affb9e", - "metadata": {}, + "id": "bf06f54c", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import os\n", @@ -1676,8 +1928,10 @@ }, { "cell_type": "markdown", - "id": "70698b7b", - "metadata": {}, + "id": "9e346aac", + "metadata": { + "editable": true + }, "source": [ "## Another Example, now with a polynomial fit" ] @@ -1685,8 +1939,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "d5e411b4", - "metadata": {}, + "id": "b3ce165d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import os\n", @@ -1771,8 +2028,10 @@ }, { "cell_type": "markdown", - "id": "10b0a7eb", - "metadata": {}, + "id": "6c6d6c09", + "metadata": { + "editable": true + }, "source": [ "## To think about, first part\n", "\n", @@ -1797,8 +2056,10 @@ }, { "cell_type": "markdown", - "id": "0fdacb5f", - "metadata": {}, + "id": "6a80380c", + "metadata": { + "editable": true + }, "source": [ "## More thinking\n", "\n", @@ -1830,8 +2091,10 @@ }, { "cell_type": "markdown", - "id": "6ade509a", - "metadata": {}, + "id": "1f6d0596", + "metadata": { + "editable": true + }, "source": [ "## Still thinking\n", "\n", @@ -1842,8 +2105,11 @@ { "cell_type": "code", "execution_count": 7, - "id": "ca8d02c2", - "metadata": {}, + "id": "79e0be06", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "#Model training, we compute the mean value of y and X\n", @@ -1864,8 +2130,10 @@ }, { "cell_type": "markdown", - "id": "bb52b09b", - "metadata": {}, + "id": "525c4ef5", + "metadata": { + "editable": true + }, "source": [ "## What does centering (subtracting the mean values) mean mathematically?\n", "\n", @@ -1878,8 +2146,10 @@ }, { "cell_type": "markdown", - "id": "3cde4c58", - "metadata": {}, + "id": "4585e4a1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n", @@ -1888,8 +2158,10 @@ }, { "cell_type": "markdown", - "id": "e0e6cac8", - "metadata": {}, + "id": "ba6207c3", + "metadata": { + "editable": true + }, "source": [ "Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.\n", "\n", @@ -1901,8 +2173,10 @@ }, { "cell_type": "markdown", - "id": "1e631321", - "metadata": {}, + "id": "81c29129", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n", @@ -1911,16 +2185,20 @@ }, { "cell_type": "markdown", - "id": "8e5038d9", - "metadata": {}, + "id": "67596e4b", + "metadata": { + "editable": true + }, "source": [ "for all $j$. For $\\beta_0$ we have" ] }, { "cell_type": "markdown", - "id": "bb56c7ca", - "metadata": {}, + "id": "9bdb9d87", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n", @@ -1929,16 +2207,20 @@ }, { "cell_type": "markdown", - "id": "3ed4442a", - "metadata": {}, + "id": "cb81ff47", + "metadata": { + "editable": true + }, "source": [ "Multiplying away the constant $2/n$, we obtain" ] }, { "cell_type": "markdown", - "id": "9104f8b7", - "metadata": {}, + "id": "71a389fc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n", @@ -1947,8 +2229,10 @@ }, { "cell_type": "markdown", - "id": "4a12287c", - "metadata": {}, + "id": "fa4b9339", + "metadata": { + "editable": true + }, "source": [ "## Further Manipulations\n", "\n", @@ -1958,8 +2242,10 @@ }, { "cell_type": "markdown", - "id": "5b425023", - "metadata": {}, + "id": "a31e16e7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n", @@ -1968,16 +2254,20 @@ }, { "cell_type": "markdown", - "id": "e6fa30b0", - "metadata": {}, + "id": "5d6c27e0", + "metadata": { + "editable": true + }, "source": [ "We obtain then" ] }, { "cell_type": "markdown", - "id": "28be70e6", - "metadata": {}, + "id": "6e855d2b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", @@ -1986,16 +2276,20 @@ }, { "cell_type": "markdown", - "id": "2b41476d", - "metadata": {}, + "id": "ac3b2e85", + "metadata": { + "editable": true + }, "source": [ "If we define" ] }, { "cell_type": "markdown", - "id": "97b05bfa", - "metadata": {}, + "id": "d677c104", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mu_1=\\frac{1}{n}\\sum_{i=0}^{n-1} (X_{i1},\n", @@ -2004,16 +2298,20 @@ }, { "cell_type": "markdown", - "id": "bbb7f94a", - "metadata": {}, + "id": "b7d4987d", + "metadata": { + "editable": true + }, "source": [ "and if we define the mean value of the outputs as" ] }, { "cell_type": "markdown", - "id": "c0d06b59", - "metadata": {}, + "id": "f610e10c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n", @@ -2022,16 +2320,20 @@ }, { "cell_type": "markdown", - "id": "da1e1b48", - "metadata": {}, + "id": "592c0ada", + "metadata": { + "editable": true + }, "source": [ "we have" ] }, { "cell_type": "markdown", - "id": "b840be0b", - "metadata": {}, + "id": "95eb9032", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0 = \\mu_y - \\beta_1\\mu_{1}.\n", @@ -2040,16 +2342,20 @@ }, { "cell_type": "markdown", - "id": "c034bf6b", - "metadata": {}, + "id": "6d40a325", + "metadata": { + "editable": true + }, "source": [ "In the general case, that is we have more parameters than $\\beta_0$ and $\\beta_1$, we have" ] }, { "cell_type": "markdown", - "id": "8f64ec46", - "metadata": {}, + "id": "a73ec1dc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n", @@ -2058,16 +2364,20 @@ }, { "cell_type": "markdown", - "id": "dea8f0e3", - "metadata": {}, + "id": "110f580a", + "metadata": { + "editable": true + }, "source": [ "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)" ] }, { "cell_type": "markdown", - "id": "7d83c4bd", - "metadata": {}, + "id": "df442fd7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n", @@ -2076,8 +2386,10 @@ }, { "cell_type": "markdown", - "id": "bff682c6", - "metadata": {}, + "id": "4ae3df8b", + "metadata": { + "editable": true + }, "source": [ "## Wrapping it up\n", "\n", @@ -2086,8 +2398,10 @@ }, { "cell_type": "markdown", - "id": "be879f0d", - "metadata": {}, + "id": "145ac8a9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", @@ -2096,8 +2410,10 @@ }, { "cell_type": "markdown", - "id": "c6f4c9e0", - "metadata": {}, + "id": "2ebb35d5", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n", "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n", @@ -2107,8 +2423,10 @@ }, { "cell_type": "markdown", - "id": "d82a7342", - "metadata": {}, + "id": "3f2293ec", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", @@ -2117,16 +2435,20 @@ }, { "cell_type": "markdown", - "id": "2917e850", - "metadata": {}, + "id": "73504771", + "metadata": { + "editable": true + }, "source": [ "What does this mean? And why do we insist on all this? Let us look at some examples." ] }, { "cell_type": "markdown", - "id": "50b08ee1", - "metadata": {}, + "id": "3d935fba", + "metadata": { + "editable": true + }, "source": [ "## Linear Regression code, Intercept handling first\n", "\n", @@ -2137,8 +2459,11 @@ { "cell_type": "code", "execution_count": 8, - "id": "2c599d3e", - "metadata": {}, + "id": "c109b1b5", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -2231,8 +2556,10 @@ }, { "cell_type": "markdown", - "id": "b27a10b8", - "metadata": {}, + "id": "534b2f34", + "metadata": { + "editable": true + }, "source": [ "The intercept is the value of our output/target variable\n", "when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case). \n", @@ -2250,8 +2577,10 @@ }, { "cell_type": "markdown", - "id": "b217703e", - "metadata": {}, + "id": "622d039d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n", @@ -2260,16 +2589,20 @@ }, { "cell_type": "markdown", - "id": "04f6b527", - "metadata": {}, + "id": "c6b33efa", + "metadata": { + "editable": true + }, "source": [ "but when we take out the intercept, this equation becomes" ] }, { "cell_type": "markdown", - "id": "ca6755df", - "metadata": {}, + "id": "c184e92a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n", @@ -2278,16 +2611,20 @@ }, { "cell_type": "markdown", - "id": "3e2e804d", - "metadata": {}, + "id": "61f55115", + "metadata": { + "editable": true + }, "source": [ "For Lasso regression we have" ] }, { "cell_type": "markdown", - "id": "b28a22e4", - "metadata": {}, + "id": "4aa4ba1c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n", @@ -2296,16 +2633,20 @@ }, { "cell_type": "markdown", - "id": "b580959b", - "metadata": {}, + "id": "534a2cca", + "metadata": { + "editable": true + }, "source": [ "It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get a MSE which then contains the intercept." ] }, { "cell_type": "markdown", - "id": "0a7136a9", - "metadata": {}, + "id": "c45e132c", + "metadata": { + "editable": true + }, "source": [ "## Code Examples\n", "\n", @@ -2314,43 +2655,13 @@ }, { "cell_type": "code", - "execution_count": 2, - "id": "fb95e6e4", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Beta values for own Ridge implementation\n", - "[ 1.04152388 -0.08796629 -0.2265552 -0.06821103 0.04383751 0.08561268\n", - " 0.08570563 0.06773487 0.04507 0.02407463 0.00721908 -0.00497338\n", - " -0.01285634 -0.01711057 -0.0184785 -0.01764665 -0.01520244 -0.01162643\n", - " -0.00730033 -0.00252065]\n", - "Beta values for Scikit-Learn Ridge implementation\n", - "[ 1.04152388 -0.08796629 -0.2265552 -0.06821103 0.04383751 0.08561268\n", - " 0.08570563 0.06773487 0.04507 0.02407463 0.00721908 -0.00497338\n", - " -0.01285634 -0.01711057 -0.0184785 -0.01764665 -0.01520244 -0.01162643\n", - " -0.00730033 -0.00252065]\n", - "MSE values for own Ridge implementation\n", - "1.621815415486309e-05\n", - "MSE values for Scikit-Learn Ridge implementation\n", - "1.6218154154878254e-05\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAERCAYAAACU1LsdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmM0lEQVR4nO3de5gV1Znv8e8vAoKiYKDHG6OgIyRcG2g0qBAdVBSVqKMHUGOQOEgSNU4iBz3J8RLHUWMmQXOBId6SyKCj8S4SjRHhREUuIqKIihAFUVsSFVCQhvf8UdWVzXb3vXc3jb/P8/RD7VWrVr273Na7q1bttRQRmJmZAXyhuQMwM7Mdh5OCmZllnBTMzCzjpGBmZhknBTMzyzgpmJlZpkUmBUm3SnpP0tJGam+WpA8kPZxXPlfS4vTvbUn3N3aMko5K239J0lNp2T9KelLSy2n5d3Pqn5GWbZNUllPeWtJvJL0oaZmky2qKRdKVktbkvMcReesPkLRB0iX1bUtSV0mf5JRPre0xrOGYfZjT5uUNbdPMEi0yKQC3A8c3Yns3AF/PL4yIIRFRGhGlwDPAvfl1JN0u6aj6xCipI/ArYGRE9ALOSFdVAN+PiJ7AV4DvSOqZrlsKnAbMyWvuDGDXiOgDDATOl9S1FrH8rPI9RsTMvHU/BR6tw/uqqq0VOeUTqti2rubmtPmjRmrT7HOvRSaFiJgD/DW3TNLB6Tf+hek3/C/Vob0ngPVVrZe0J/DPwP0NibGAM4F7I+LNdJv30n/XRsSidHk9sAzYP329LCKWF9olsLukVkA74FPgozrEsh1JpwArgZfq8b5qu4/jJD0jaZGkuyW1b4x2zaz+WmRSqMI04MKIGAhcQvINvLGcAjwRER81YpsA3YG9JM1Ok9k5+RXSb/v9gXk1tHUPsBFYC7wJ/CQianPyvkDSkvS20F7pPtsDk4Crav9WCreV6ibpeUlPSRqS7qMz8EPgmIgYACwAvleHfQ2W9IKkRyX1qmOcZlaFVs0dQGNIT2KHA3dLqizeNV13GlDo9sKaiBhey12MAW7O2d9w4Pr05QHAkZI2AJsj4rA6hN6K5FbPMJJv989IejYiXs15X78HLq5FQjoU2ArsB+wFzJX0x4h4o5ptpgBXk1xlXA38JzAOuJLkVtCGnONZk6raWgscEBHrJA0E7k9P4l8BegJ/TvfRhuQWHZKuBU4usI/7I+KHwCLgwDS+ESRXcIfUNlAzq9pOkRRIrng+SO/9byci7qVAX0Btpd9oDwVOzWnzD8Af0vW3A7dHxOx6NL8aWBcRG4GNkuYA/YBXJbUmSQjT0/dQkzOBWRGxBXhP0p+BMqDKpBAR71YuS/o1UNnRfhhwuqQfAx2BbZI2RcQv6tpWRGwGNqfLCyWtILlCEvB4RIwp0NZlwGX55TnrP8pZninpV5I6R8T7VW1jZrWzU9w+Sk8SKyWdAaBEv0Zq/nTg4YjY1Ejt5XqA5CqjlaTdSE7Gy5R8db4FWBYRP61lW2+S9HsgaXeSb+KvVLeBpH1zXp5K0old2cHeNSK6ApOB/6guIVTXlqQSSbukyweRfKN/A3gWOELSP1XGLKl7bd6opH3SY4SkQ0k+x+tqs62ZVa9FJgVJM0huNfSQtFrSN4GzgG9KeoGkc/RrdWhvLnA3MCxtL/e20mhgRiPFiKQJkiZA0mkMzAKWAM8BN0fEUuAIkqeh/rnAI56nSloNDAYekfSHdJe/BNpLegmYD9wWEUuqiwX4sZJHWJcARwP/Vt/3VU1bQ4ElkhaT9HtMiIi/RkQ5MBaYkW7zDFDbhwNOB5am/61vAkaHh/s1axTy/0tmZlapRV4pmJlZcbS4jubOnTtH165dmzsMM7MWZeHChe9HRElN9VpcUujatSsLFixo7jDMzFoUSX+pTT3fPjIzs4yTgpmZZZwUzMws0+L6FMwssWXLFlavXs2mTcX4XaW1VG3btqVLly60bt26Xts7KZi1UKtXr2aPPfaga9eu1GGMKtuJRQTr1q1j9erVdOvWrV5t+PaRWQu1adMmOnXq5IRgGUl06tSpQVePTgpmLZgTguVr6GfCScHMzDJFSwqSeuQM5rZY0keSLs6r00HSQ+lkKS9JOrdY8ZhZ45PE2Wefnb2uqKigpKSEk046CYB3332Xk046iX79+tGzZ09GjEimAV+1ahXt2rWjtLQ0+/vtb3/bpLHnxtCzZ0/OOecctmzZAsCCBQu46KKLCm7XtWtX3n+/YaO033bbbdn7btOmDX369KG0tJRLL7201m1MnjyZjz/+uEFxFBQRRf8DdgHeIZkYJbf8/wDXp8slJNM8tqmurYEDB4aZRbz88svNHULsvvvu0a9fv/j4448jImLmzJnRr1+/OPHEEyMiYvz48TF58uSs/gsvvBAREStXroxevXo1fcA5cmOoqKiIo48+Ou64444atzvwwAOjvLy80eKob3vVbVfoswEsiFqcr5vq9tEwksnb839mHcAe6dj47dOkUNFEMZlZIxgxYgSPPPIIADNmzGDMmL/Pm7R27Vq6dOmSve7bt2+d2n7iiSfo378/ffr0Ydy4cWzevJn58+dz2mmnAfDAAw/Qrl07Pv30UzZt2sRBBx0EwFFHHcWkSZM49NBD6d69O3Pnzq12P7vssguHHnooa9asAWD27NnZ1c66des47rjj6NWrF+edd17lF1oArr76anr06MGRRx7JmDFj+MlPfgLAihUrOP744xk4cCBDhgzhlVeqndokc8MNNzBo0CD69u3LFVdcAcDGjRs58cQT6devH7179+auu+7ipptu4u233+boo4/m6KOPrsMRrVlTPZJa1ZwEvwAeBN4G9gBGRcS2JorJbOdx8cWweHHjtllaCpMn11ht9OjR/OhHP+Kkk05iyZIljBs3LjsJf+c732HUqFH84he/4JhjjuHcc89lv/32A5ITZ2lpadbOz3/+c4YMGZK93rRpE2PHjuWJJ56ge/funHPOOUyZMoULLriAxel7nTt3Lr1792b+/PlUVFRw2GF/nw23oqKC5557jpkzZ3LVVVfxxz/+scr3sGnTJubNm8eNN974mXVXXXUVRx55JJdffjmPPPIIt9xyCwDz58/n97//PS+88AJbtmxhwIABDBw4EIDx48czdepUDjnkEObNm8e3v/1t/vSnP1V7HB977DFee+01nnvuOSKCkSNHMmfOHMrLy9lvv/2yxPvhhx/SoUMHfvrTn/Lkk0/SuXPnatutq6InBUltgJEUnl5xOLCYZMawg4HHJc2NvPmIJY0HxgMccMABRY3XzOqmb9++rFq1ihkzZmR9BpWGDx/OG2+8waxZs3j00Ufp378/S5cuBeDggw/OTu6FLF++nG7dutG9ezIh3ze+8Q1++ctfcvHFF3PwwQezbNkynnvuOb73ve8xZ84ctm7dul1SqbyaGDhwIKtWrSq4j8rEtHLlSk488cSCVzJz5szh3nuTGXFPPPFE9tprLwD+/Oc/87WvfY22bdvStm1bTj45mVZ8w4YNPP3005xxxhlZG5s3b67uEAJJUnjsscfo379/1s5rr73GkCFD+P73v8+kSZM46aSTtnuPxdAUVwonAIsiZw7fHOcC16X3u16XtJJk9q3ncitFxDRgGkBZWZlnBTLLV4tv9MU0cuRILrnkEmbPns26ddvPjPrFL36RM888kzPPPJOTTjqJOXPmZN+o62vo0KE8+uijtG7dmmOOOYaxY8eydetWbrjhhqzOrrvuCiS3hioqCt+VrkxM77//PkcccQQPPvggI0eObFBs27Zto2PHjtUmvEIigssuu4zzzz//M+sWLVrEzJkz+eEPf8iwYcO4/PLLGxRjdZqiT2EMVU9n+SZJfwOS9gZ6UM1E82a2Yxo3bhxXXHEFffr02a78T3/6U/aEzPr161mxYkWtr/Z79OjBqlWreP311wH43e9+x1e/+lUAhgwZwuTJkxk8eDAlJSWsW7eO5cuX07t373rF37lzZ6677jquvfbaz6wbOnQo//3f/w3Ao48+yt/+9jcAjjjiCB566CE2bdrEhg0bePjhhwHYc8896datG3fffTeQnOxfeOGFGmMYPnw4t956Kxs2bABgzZo1vPfee7z99tvstttunH322UycOJFFixYBsMcee7B+/fp6vd/qFPVKIZ1A/ljg/JyyyvmJpwJXA7dLehEQMCkiGvasl5k1uS5duhR8hHPhwoVccMEFtGrVim3btnHeeecxaNAgVq1a9Zk+hXHjxm3XRtu2bbnttts444wzqKioYNCgQUyYMAGAww47jHfffZehQ4cCyS2sd955p0E/3DrllFO48sorP9MpfcUVVzBmzBh69erF4YcfniW1QYMGMXLkSPr27cvee+9Nnz596NChAwDTp0/nW9/6Fv/+7//Oli1bGD16NP369at2/8cddxzLli1j8ODBALRv35477riD119/nYkTJ/KFL3yB1q1bM2XKFCDptzj++OPZb7/9ePLJJ+v9vvO1uDmay8rKwpPsmMGyZcv48pe/3NxhfK5t2LCB9u3b8/HHHzN06FCmTZvGgAEDmjusgp8NSQsjoqymbT0gnplZPY0fP56XX36ZTZs28Y1vfGOHSAgN5aRgZlZPlX0NOxOPfWRmZhknBTMzyzgpmJlZxknBzMwyTgpmVm/FHjr7qKOOokePHvTr149BgwZt9yvhESNG8MEHH3xmmyuvvDIbmK6+1q1bl8W1zz77sP/++2evP/3001q1MXv2bJ5++ukGxdEc/PSRmdXb7rvvztKlS/nkk09o164djz/+OPvvv3+2/vLLL+fYY4/lu9/9LgBLlizJ1tU09lGl6dOnU1ZWxm233cbEiRN5/PHHAZg5c2bjvpkcnTp1ymK78sorad++PZdcckmd2pg9ezbt27fn8MMPL0KExeMrBTNrkGIOnZ1r8ODB2dDWsP1kN9dccw3du3fnyCOPZPny5Vmd+fPn07dvX0pLS5k4cWI2DMbWrVuZOHFiNkz1f/3Xf9UqhoULF/LVr36VgQMHMnz4cNauXQvATTfdRM+ePenbty+jR49m1apVTJ06lZ/97GeUlpbWOHT3jsRXCmY7gYtnXczidxY3apul+5Qy+fjJNdYr1tDZ+WbNmsUpp5zymfKFCxdy5513snjxYioqKrYbwvrcc8/l17/+NYMHD95uVrNbbrmFDh06MH/+fDZv3swRRxzBcccdR7du3arc/5YtW7jwwgt54IEHKCkp4a677uIHP/gBt956K9dddx0rV65k11135YMPPqBjx45MmDChXlcYzc1JwcwapFhDZ1c666yz+PTTT9mwYUPB+nPnzuXUU09lt912A8hGOf3ggw9Yv359NpbQmWeemQ1a99hjj7FkyRLuueceIJmj4LXXXqs2KSxfvpylS5dy7LHHAsnVxr777psdg7POOotTTjmlYOJqSZwUzHYCtflGX0zFHDp7+vTpDBw4kIkTJ3LhhRdmcxs0RETw85//nOHDh9dpm169evHMM898Zt0jjzzCnDlzeOihh7jmmmt48cUXGxxjc3Gfgpk1WDGGzs4liauvvppnn332M1NbDh06lPvvv59PPvmE9evX89BDDwHQsWNH9thjD+bNmwfAnXfemW0zfPhwpkyZwpYtWwB49dVX2bhxY7Ux9OjRg/Ly8iwpbNmyhZdeeolt27bx1ltvcfTRR3P99dfz4YcfsmHDhqINbV1sTgpm1mDVDZ1dVlZG3759GTx4cDZ0Nvy9T6Hy76abbqp2H+3ateP73//+dhPpAAwYMIBRo0bRr18/TjjhhKx9SPoO/vVf/5XS0lI2btyYDW193nnn0bNnTwYMGEDv3r05//zzq5yIp1KbNm245557mDRpEv369aO0tJSnn36arVu3cvbZZ9OnTx/69+/PRRddRMeOHTn55JO57777WlxHs4fONmuhPHR2zSqHtga47rrrWLt2bcF5mHc2HjrbzKyARx55hGuvvZaKigoOPPBAbr/99uYOaYfnpGBmO61Ro0YxatSo5g6jRSlan4KkHpIW5/x9JOniAvWOSte/JOmpYsVjtjNqabd/rfga+pko2pVCRCwHSgEk7QKsAe7LrSOpI/Ar4PiIeFPSPxQrHrOdTdu2bVm3bh2dOnVq0NzEtvOICNatW0fbtm3r3UZT3T4aBqyIiL/klZ8J3BsRbwJExHtNFI9Zi9elSxdWr15NeXl5c4diO5C2bdtuN7RIXTVVUhgNzChQ3h1oLWk2sAdwY0R8dqhEM/uM1q1bV/sLXLP6KHpSkNQGGAlcVsX+B5JcSbQDnpH0bES8mtfGeGA8UK8fvpiZWe00xY/XTgAWRcS7BdatBv4QERsj4n1gDtAvv1JETIuIsogoKykpKXK4ZmafX02RFMZQ+NYRwAPAkZJaSdoNOAxY1gQxmZlZAUW9fSRpd+BY4PycsgkAETE1IpZJmgUsAbYBN0fE0mLGZGZmVStqUoiIjUCnvLKpea9vALYfzMTMzJqFB8QzM7OMk4KZmWWcFMzMLOOkYGZmGScFMzPLOCmYmVnGScHMzDJOCmZmlnFSMDOzjJOCmZllnBTMzCzjpGBmZhknBTMzyzgpmJlZxknBzMwyTgpmZpZxUjAzs4yTgpmZZYqWFCT1kLQ45+8jSRdXUXeQpApJpxcrHjMzq1nR5miOiOVAKYCkXYA1wH359dJ11wOPFSsWMzOrnaa6fTQMWBERfymw7kLg98B7TRSLmZlVoamSwmhgRn6hpP2BU4Ep1W0sabykBZIWlJeXFylEMzMrelKQ1AYYCdxdYPVkYFJEbKuujYiYFhFlEVFWUlJShCjNzAyK2KeQ4wRgUUS8W2BdGXCnJIDOwAhJFRFxfxPEZWZmeZoiKYyhwK0jgIjoVrks6XbgYScEM7PmU9TbR5J2B44F7s0pmyBpQjH3a2Zm9VPUK4WI2Ah0yiubWkXdscWMxczMauZfNJuZWcZJwczMMk4KZmaWcVIwM7OMk4KZmWWcFMzMLOOkYGZmGScFMzPLOCmYmVnGScHMzDJOCmZmlnFSMDOzjJOCmZllnBTMzCzjpGBmZhknBTMzyzgpmJlZpmhJQVIPSYtz/j6SdHFenbMkLZH0oqSnJfUrVjxmZlazok3HGRHLgVIASbsAa4D78qqtBL4aEX+TdAIwDTisWDGZmVn1ijpHc45hwIqI+EtuYUQ8nfPyWaBLE8VjZmYFNFWfwmhgRg11vgk8WmiFpPGSFkhaUF5e3ujBmZlZouhJQVIbYCRwdzV1jiZJCpMKrY+IaRFRFhFlJSUlxQnUzMya5PbRCcCiiHi30EpJfYGbgRMiYl0TxGNmZlVoittHY6ji1pGkA4B7ga9HxKtNEIuZmVWjqFcKknYHjgXOzymbABARU4HLgU7AryQBVEREWTFjMjOzqhU1KUTERpKTfm7Z1Jzl84DzihmDmZnVnn/RbGZmGScFMzPLOCmYmVmm2qQg6eyc5SPy1l1QrKDMzKx51HSl8L2c5Z/nrRvXyLGYmVkzqykpqIrlQq/NzKyFqykpRBXLhV6bmVkLV9PvFL4kaQnJVcHB6TLp64OKGpmZmTW5mpLCl5skCjMz2yFUmxTy5z+Q1AkYCrwZEQuLGZiZmTW9mh5JfVhS73R5X2ApyVNHv8ufWtPMzFq+mjqau0XE0nT5XODxiDiZZMpMP5JqZraTqSkpbMlZHgbMBIiI9cC2YgVlZmbNo6aO5rckXQisBgYAswAktQNaFzk2MzNrYjVdKXwT6AWMBUZFxAdp+VeA24oXlpmZNYeanj56D5hQoPxJ4MliBWVmZs2j2qQg6cHq1kfEyMYNx8zMmlNNfQqDgbdI5lieRx3GO5LUA7grp+gg4PKImJxTR8CNwAjgY2BsRCyq7T7MzKxx1ZQU9iGZY3kMcCbwCDAjIl6qqeGIWA6UAkjaBVgD3JdX7QTgkPTvMGBK+q+ZmTWDajuaI2JrRMyKiG+QdC6/Dsyux1wKw4AV+b+QBr4G/DYSzwId0x/JmZlZM6jpSgFJuwInklwtdAVu4rPf+GsymuQWVL79SW5PVVqdlq3Ni2E8MB7ggAMOqOOuzcystmrqaP4t0JvkR2tX5fy6udYktQFGApfVK0IgIqYB0wDKyso8ZLeZWZHUdKVwNrAR+C5wUdIvDCQdzhERe9ZiHycAiyLi3QLr1gD/mPO6S1pmZmbNoKbfKdT047baGEPhW0cADwIXSLqTpIP5w4hYW0VdMzMrshr7FBpC0u4kTy+dn1M2ASAippLclhpB0oH9Mcmge2Zm1kyKmhQiYiPQKa9sas5yAN8pZgxmZlZ7jXF7yMzMdhJOCmZmlnFSMDOzjJOCmZllnBTMzCzjpGBmZhknBTMzyzgpmJlZxknBzMwyTgpmZpZxUjAzs4yTgpmZZZwUzMws46RgZmYZJwUzM8s4KZiZWcZJwczMMkVNCpI6SrpH0iuSlkkanLe+g6SHJL0g6SVJno7TzKwZFXU6TuBGYFZEnC6pDbBb3vrvAC9HxMmSSoDlkqZHxKdFjsvMzAooWlKQ1AEYCowFSE/0+Sf7APaQJKA98FegolgxmZlZ9Yp5+6gbUA7cJul5STdL2j2vzi+ALwNvAy8C342IbfkNSRovaYGkBeXl5UUM2czs862YSaEVMACYEhH9gY3ApXl1hgOLgf2AUuAXkvbMbygipkVEWUSUlZSUFDFkM7PPt2ImhdXA6oiYl76+hyRJ5DoXuDcSrwMrgS8VMSYzM6tG0ZJCRLwDvCWpR1o0DHg5r9qbaTmS9gZ6AG8UKyYzM6tesZ8+uhCYnj559AZwrqQJABExFbgauF3Si4CASRHxfpFjMjOzKhQ1KUTEYqAsr3hqzvq3geOKGYOZmdWef9FsZmYZJwUzM8s4KZiZWcZJwczMMk4KZmaWcVIwM7OMk4KZmWWcFMzMLOOkYGZmGScFMzPLOCmYmVnGScHMzDJOCmZmlnFSMDOzjJOCmZllnBTMzCzjpGBmZpmiJgVJHSXdI+kVScskDS5Q5yhJiyW9JOmpYsZjZmbVK/YczTcCsyLi9HSe5t1yV0rqCPwKOD4i3pT0D0WOx8zMqlG0pCCpAzAUGAsQEZ8Cn+ZVOxO4NyLeTOu8V6x4zMysZsW8fdQNKAduk/S8pJsl7Z5Xpzuwl6TZkhZKOqdQQ5LGS1ogaUF5eXkRQzYz+3wrZlJoBQwApkREf2AjcGmBOgOBE4HhwP+V1D2/oYiYFhFlEVFWUlJSxJDNzD7fipkUVgOrI2Je+voekiSRX+cPEbExIt4H5gD9ihiTmZlVo2hJISLeAd6S1CMtGga8nFftAeBISa0k7QYcBiwrVkxmZla9Yj99dCEwPX3y6A3gXEkTACJiakQskzQLWAJsA26OiKVFjsnMzKqgiGjuGOqkrKwsFixY0NxhmJm1KJIWRkRZTfX8i2YzM8s4KZiZWcZJwczMMk4KZmaWcVIwM7OMk4KZmWWcFMzMLOOkYGZmGScFMzPLOCmYmVnGScHMzDJOCmZmlnFSMDOzjJOCmZllnBTMzCzjpGBmZhknBTMzyxQ1KUjqKOkeSa9IWiZpcBX1BkmqkHR6MeMxM7PqFXuO5huBWRFxejpP8275FSTtAlwPPFbkWMzMrAZFu1KQ1AEYCtwCEBGfRsQHBapeCPweeK9YsZiZWe0U8/ZRN6AcuE3S85JulrR7bgVJ+wOnAlOqa0jSeEkLJC0oLy8vXsRmZp9zxUwKrYABwJSI6A9sBC7NqzMZmBQR26prKCKmRURZRJSVlJQUJVgzMytun8JqYHVEzEtf38Nnk0IZcKckgM7ACEkVEXF/EeMyM7MqFC0pRMQ7kt6S1CMilgPDgJfz6nSrXJZ0O/CwE4KZWfMp9tNHFwLT0yeP3gDOlTQBICKmFnnfZmZWR0VNChGxmOQWUa6CySAixhYzFjMzq5l/0WxmZhknBTMzyzgpmJlZxknBzMwyTgpmZpZxUjAzs4yTgpmZZZwUzMws46RgZmYZJwUzM8s4KZiZWcZJwczMMk4KZmaWcVIwM7OMk4KZmWWcFMzMLKOIaO4Y6kRSOfCXem7eGXi/EcNpLDtqXLDjxua46sZx1c3OGNeBEVFSU6UWlxQaQtKCiMifCa7Z7ahxwY4bm+OqG8dVN5/nuHz7yMzMMk4KZmaW+bwlhWnNHUAVdtS4YMeNzXHVjeOqm89tXJ+rPgUzM6ve5+1KwczMquGkYGZmmZ0iKUi6QdIrkpZIuk9Sxyrq3SrpPUlL88q/KOlxSa+l/+6VlkvSTZJeT9seUKS4jpe0PN3PpTnlcyUtTv/elnR/Wn6UpA9z1l3exHHdLmllzv5L0/LmPl7T0/Kl6X/r1ml5cx+vbpLmpeV3SWqTlu+avn49Xd+1jnGdIeklSdskVfmYoqTvpsfkJUkX55TflXNMVklanJZ3lfRJzrqpTRzXlZLW5Ox/RM66y9LjtVzS8CaOq+DnYAc4Xo17/oqIFv8HHAe0SpevB66vot5QYACwNK/8x8Cl6fKlldsDI4BHAQFfAeY1dlzALsAK4CCgDfAC0LNAvd8D56TLRwEPF/N4VRcXcDtweoFtmvV4pftX+jcD+NYOcrz+BxidLk/NievbwNR0eTRwVx3j+jLQA5gNlFVRpzewFNgNaAX8EfinAvX+E7g8Xe6a//9IU8YFXAlcUmCbnulx3RXolh7vXZowroKfgx3geDXq+WunuFKIiMcioiJ9+SzQpYp6c4C/Flj1NeA36fJvgFNyyn8biWeBjpL2beS4DgVej4g3IuJT4M50vxlJewL/DNxf2303RVwFNOvxioiZ6b4DeK6K7eusIXFJEsl/u3vSevmfr8rP3T3AsLR+beNaFhHLa6j2ZZKTwcfpe3gKOC23QrrP/0WSSBusseIq4GvAnRGxOSJWAq+THPcmiau255m6aoTj1ajnr50iKeQZR5Id62LviFibLr8D7J0u7w+8lVNvdVrWmHHVZh+nAE9ExEc5ZYMlvSDpUUm96hlTQ+K6Jr0k/ZmkXWu5TVPERXrb6OvArJzi5jpenYAPck4mufFm26TrP0zrN6alwBBJnSTtRvLt8R/z6gwB3o2I13LKukl6XtJTkoY0cky1ieuC9PN1a+XtEBr381XfuCrlfw6a83g16vmrVePEW3yS/gjsU2DVDyLigbTOD4AKYHp99xMRIanWz+k2UVxjgJtzXi8iGcdkQ3q/9X7gkCaM6zKSD18bkuemJwE/qs2GTXS8fgXMiYi56evmPl71Vpu4qhMRyyRdDzwGbAQWA1vzqo1h+6uEtcABEbFO0kDgfkm9cr+UFDmuKcDVQKT//ifJSbhGTXG8CnwOmvt45dar0/mrkBaTFCLimOrWSxoLnAQMS28f1MW7kvaNiLXp5dV7afkatv+W0CUta8y4qt2HpM4kl8in5uzzo5zlmZJ+JalzRLyfU160uHK+lWyWdBtwSW3eS7HjSre/AigBzs/ZZ3Mer3Ukl+2t0quB3Hgrt1ktqRXQIa2fqSmu2oiIW4Bb0vfxHyTfGCvfVyuS2xADc+pvBjanywslrQC6AwuaIq6IeDcnvl8DD6cvG/z5akhc6eux5H0Omvt40YDzV1U7avF/wPHAy0BJLep25bMdzTewfUfNj9PlE9m+o+a5xo6LJDG/QdJxVtlB2Stn/QTgN3nb7MPff3h4KPBm5eumiAvYN/1XwGTguh3heAHnAU8D7Xaw43U323c0fztd/g7bdzT/Tz0//7OpooMyXf8P6b8HAK8AHfPe21N59UtIO3BJOs7XAF9sqrgqP1/p8r+R9CMA9GL7juY3qENHcyPEVfBzsAMcr0Y9f9X5A7gj/pF0OL1Fckm1OOd/tP2AmTn1ZpBc6m0hybLfTMs7AU8Ar5H06n8xLRfwS5KnHF6s7j9YA+MaAbya7ucHBT4ox+eVXQC8lP4P8ixweFPGBfwpPR5LgTuA9jvC8SK5pF+Rs33l0zTNfbwOIun4fp0kQeyalrdNX7+erj+ojnGdmn6ONwPvAn+oIq65JCezF0i+4ea2cTswIa/sX9LjtZjk1tvJTRkX8Lv087MEeJDtk8QP0uO7HDihieOq6nPQ3MerUc9fHubCzMwyO+PTR2ZmVk9OCmZmlnFSMDOzjJOCmZllnBTMzCzjpGAtmqQNDdj2gnQEyUh/JFhZXuXokpL2lfRwunxU5XJDSZqtakbIzKm3KjfWKur8MWdoCLM6cVKwz7M/A8cAf8krP4FkGIxDgPEkwy5U+h7w6yaJrv5+RzICq1mdOSnYTiH9dn+DkvHmX5Q0Ki3/QjqsxStKxpqfKel0gIh4PiJWFWiuutEl/4XtB9qr3P+hkp5JB0V7WlKPtHyspPvTfa9Kr06+l9Z7VtIXc5r5upLx+JdKOjTdvpOkx5SMoX8zyQ+SKvd5v6SF6brxOe08SDKekVmdOSnYzuI0oBToR/Lt/4b0RH4aydAmPUlGTh1ci7YKji4pqRvwt0jGusn3CjAkIvoDlwP/kbOudxrHIOAa4OO03jPAOTn1douIUpJv+bemZVcA/y8iegH3kQxxUGlcRAwEyoCLJHUCiIi/AbtWvjarixYzIJ5ZDY4EZkTEVpIBwp4iOQkfCdwdEduAdyQ92YB97AuUV7GuA/AbSYeQjO7ZOmfdkxGxHlgv6UPgobT8RaBvTr0ZkMz7IWlPJTN7DeXv4/k/IulvOfUvklQ5UOI/ktzuqhxQ7z2SYRK2G2DPrCa+UjD7rKpGl/yEZLyiQq4mOfn3Bk7Oq5d7ZbEt5/U2tv9ilj/mTJVj0Eg6iuSKaHBE9AOez9tn2zReszpxUrCdxVxglKRdJJWQfMN+jqQz+V/SvoW9SabmrMmDwDlpP8VXgA8jGS78VZJbUYV04O/DEo+t53uo7Ac5Mt3nh8Ac4My0/ASg8qmiDiS3sj6W9CWSUTBJ64lkZNhV9YzDPsecFGxncR/JqJovkIzi+r8j4h2Sua1Xk4wueQfJKJYfAki6SNJqkiuBJWlHLsBMkmGZXyd50ujbABGxEVgh6Z8K7P/HwLWSnqf+t2U3pdtPBb6Zll0FDJX0EsltpDfT8llAK0nLgOtIRn+tNBB4Nv4+25tZrXmUVNvpSWofyaxrnUiuHo5IE0Z92joVGBgRP2zUIBuRpBuBByPiieaOxVoedzTb58HDaadtG+Dq+iYEgIi4rwU81bPUCcHqy1cKZmaWcZ+CmZllnBTMzCzjpGBmZhknBTMzyzgpmJlZ5v8DkpQeAPsEM8MAAAAASUVORK5CYII=\n", - "text/plain": [ - "
    " - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "execution_count": 9, + "id": "63d15269", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", @@ -2382,10 +2693,10 @@ "p = Maxpolydegree\n", "I = np.eye(p,p)\n", "# Decide which values of lambda to use\n", - "nlambdas = 1\n", + "nlambdas = 6\n", "MSEOwnRidgePredict = np.zeros(nlambdas)\n", "MSERidgePredict = np.zeros(nlambdas)\n", - "lambdas = np.logspace(-2, 2, nlambdas)\n", + "lambdas = np.logspace(-4, 2, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", " OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", @@ -2421,8 +2732,10 @@ }, { "cell_type": "markdown", - "id": "345ca257", - "metadata": {}, + "id": "62fa350a", + "metadata": { + "editable": true + }, "source": [ "The results here agree when we force **Scikit-Learn**'s Ridge function to include the first column in our design matrix.\n", "We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.\n", @@ -2432,55 +2745,23 @@ }, { "cell_type": "markdown", - "id": "0c591f94", - "metadata": {}, + "id": "d4d6005b", + "metadata": { + "editable": true + }, "source": [ "## Taking out the mean" ] }, { "cell_type": "code", - "execution_count": 3, - "id": "d9dbc988", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Beta values for own Ridge implementation\n", - "[-0.11987476 -0.17941937 -0.06692651 0.02707229 0.07073126 0.07820093\n", - " 0.06661455 0.04763463 0.02786784 0.01059519 -0.00283875 -0.01213556\n", - " -0.01751971 -0.01944576 -0.01844105 -0.0150256 -0.00967471 -0.00280481\n", - " 0.00522883]\n", - "Beta values for Scikit-Learn Ridge implementation\n", - "[-0.11987476 -0.17941937 -0.06692651 0.02707229 0.07073126 0.07820093\n", - " 0.06661455 0.04763463 0.02786784 0.01059519 -0.00283875 -0.01213556\n", - " -0.01751971 -0.01944576 -0.01844105 -0.0150256 -0.00967471 -0.00280481\n", - " 0.00522883]\n", - "Intercept from own implementation:\n", - "1.0475047612229758\n", - "Intercept from Scikit-Learn Ridge implementation\n", - "1.0475047612229833\n", - "MSE values for own Ridge implementation\n", - "4.398879605532181e-05\n", - "MSE values for Scikit-Learn Ridge implementation\n", - "4.398879605534439e-05\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAERCAYAAACU1LsdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlGElEQVR4nO3deZwU1bn/8c8jIMOiEGBiRNRBoyjLMMiAIkIAQQmMuERvGFdELy54cYle9ZdEXBMVr3JdEzWIWxCjEQkBggsEo1cQlE2WCIqKqCwqAckgA8/vjzpTNs3M9Gw9wwzf9+vVL6rPOXXq6Zqmnj5V3afM3REREQHYp6YDEBGRPYeSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISKxWJgUzG2dm68xsSRX1N93MvjGzKUnlb5jZgvBYa2aTytlvPTN7L7nfhPpLzWxx6P8fZtY+lO9rZk+EuoVm1idhnfxQvijE3SqU55jZ26GveWbWPZRfl/AalpjZDjNrEequDGXvm9lVSbH9l5ktD3V3J5TfaGYrzWyFmZ2cUL464bXMSyi/2cw+S4hhUHn2YQn7bbyZfZTQZ05l+xSRwN1r3QPoDRwDLKmi/k4ETgGmlNLmReD8YsrHA31KWOca4I8l9Qvsn7A8BJgelkcCT4TlHwLziRJ4fWAd0CrU3Q3cHJZnAD8Ny4OAWcVs7xTg9bDcEVgCNA79vgr8ONT1Dc8bFsUQ/m0PLAQaAm2BVUC9ULe6KK6kbd4MXFvFf//xwJk1/T7UQ4+6+KiVIwV3nw18lVhmZoeHT87zwyf8o8rR32vA5pLqzWx/oB8wqax9mlkbYDDweCnb/VfC0yZA0S8J2wOvhzbrgG+AXMDCo4mZGbA/sLaou/AcoFlCeaJ8YEJYPhqY4+5b3b0Q+DtwRqi7DLjT3bclxABwKvCcu29z94+AlUD3kvdCycIoaoyZvRNGPZdUpB8RqVq1MimU4FHgv9y9K3At8HAV9n0a8FrSQTyVscB/AztLa2RmI81sFdGn/lGheCEwxMzqm1lboCtwsLtvJzpgLyY66LcH/hDWuQoYY2afAvcANyZtpzEwkGjEA9EooZeZtQx1g4CDQ92RoW6Omf3dzLqF8oOATxO6XRPKIEpKM0JSHpH0Mq8IB/5xZvaDUHYRsMnduwHdgP8Mr7Ws7gh93mdmDcuxnoiUok4kBTNrChwP/MnMFgC/Bw4MdWeE8+bJj7+VYxOJn7Axs5OLzmcTnfZ5PDyfE+rzgHXuPj9Vx+7+kLsfDlwP/CoUjyM64M4jSi5vATvMrAFRUugCtAYW8f3B/zLganc/GLia75NFkVOAN939q7DdZcBdRKedpgMLgB2hbX2gBXAccB3wfBiZlOYEdz8G+Ckw0sx6h/JHgMOBHOBz4H9C+UnA+WEfzgFaAkeEEURxf68lCcnpRuAoomTSIuw7EakKNX3+qqIPIItwTYHotMnnleyvD8Wc+wdaARuBjBLWG0/SNQXgt0QH9dXAF8BW4JkU29+H6JNzcXVvEY0KuhGNWIrKewNTw/ImwMKyAf9K6uMl4OxStv8b4PKwPB3om1C3CsgkOhjfmFD+N6BHMX3dTDHXEZL+Zi8CJ1fB+6DYv5seeuhRsUedGCl4dFrnIzM7C8Ainauo+zOJDjoF5YjnRndv4+5ZwFCii7vnJrczsyMSng4GPgjljc2sSVgeABS6+1LgM6C9mWWGdQYAy8LyWuAnYblfUV+hj2ah7uWk7f8w/HsI0fWEP4aqSUQXmzGzI4F9gQ3AZGComTUMp3qOAOaaWRMz2y+0b0I0ClgSnh+YsMnTi8qJEsplYfSDmR1Z9JpTKeozjF5OS+hTRCqpfk0HUBFmNoHoE2IrM1sDjAbOAR4xs18BDYDniM7Nl6W/N4hORzQN/V3k7kWnl4YCd1Zh7LcC89x9MtG59v7AduBr4ILQ7IfA38xsJ1EiOA/A3dea2S3AbDPbDnwMDAvr/Cfwv2ZWHygAEs/rnw7McPdvk8J50cxahu2PdPdvQvk4YJxFX/n9DrjA3R1438yeB5YChWGdHWZ2APBSOMNUH/iju08Pfd0dvjLqRCOnogvKjxONHN4NB/f1RAf4sng2JEYjOu11aRnXE5EUik43iIiI1I3TRyIiUjVq3emjVq1aeVZWVk2HISJSq8yfP3+Du2emalfrkkJWVhbz5s1L3VBERGJm9nFZ2qX99JGVMv+PmQ0zs/UJc9hcnO54RESkZNUxUriS6GuT+5dQP9Hdr6iGOEREJIW0jhTKMv+PiIjsOdI9UhhLNP/PfqW0+VmYEuGfRNM0fJrcIMylMwLgkEMOSUOYIrXP9u3bWbNmDQUFZf5dpewFMjIyaNOmDQ0aNKjQ+mlLConz/1jC/QCS/AWY4O7bwiyZTxL9GncX7v4o0YR35Obm6ocVIsCaNWvYb7/9yMrKIvXUVLI3cHc2btzImjVraNu2PPNLfi+dp496Es30uZro18X9zOyZxAbuvtHD9MxEp5i6pjEekTqloKCAli1bKiFIzMxo2bJlpUaPaUsKZZn/J2lenCF8P4+PiJSBEoIkq+x7otp/p5A0988oMxtCNI/OV3w/j4+IiNSAapnmwt1nuXteWL4pJISi0UQHd+/s7n3dfXl1xCMiVcPMOPfc708AFBYWkpmZSV5eHgBffvkleXl5dO7cmfbt2zNoUHSL7tWrV9OoUSNycnLix1NPPVWtsSfG0L59e84//3y2b98OwLx58xg1alSx62VlZbFhw4ZKbfuJJ56IX/e+++5Lp06dyMnJ4YYbbihzH2PHjmXr1q2ViqM4te4XzSKy52jSpAlLlizh3//+N40aNeKVV17hoIMOiutvuukmBgwYwJVXXgnAokWL4rrDDz+cBQsWVHfIuyiKYceOHQwYMIDnn3+ec845h9zcXHJzc9O23QsvvJALL7wQiJLMzJkzadWqVbn6GDt2LOeeey6NGzeu0tg0IZ6IVMqgQYP461//CsCECRPIz8+P6z7//HPatGkTP8/Ozi5X36+99hpdunShU6dODB8+nG3btvHOO+9wxhnR7cRffvllGjVqxHfffUdBQQGHHXYYAH369OH666+ne/fuHHnkkbzxxhulbqdevXp0796dzz77DIBZs2bFo52NGzdy0kkn0aFDBy6++GISZ5a+7bbbaNeuHSeccAL5+fncc889AKxatYqBAwfStWtXevXqxfLlZTsJMmbMGLp160Z2djajR48G4Ntvv2Xw4MF07tyZjh07MnHiRO6//37Wrl1L37596du3bzn2aGpKCiJ1RJ8+uz8eDncq37q1+Prx46P6DRt2ryuroUOH8txzz1FQUMCiRYs49thj47qRI0dy0UUX0bdvX+644w7Wrl0b161atWqX00fJB+6CggKGDRvGxIkTWbx4MYWFhTzyyCN06dIlHmG88cYbdOzYkXfeeYc5c+bssu3CwkLmzp3L2LFjueWWW0p9DQUFBcyZM4eBAwfuVnfLLbdwwgkn8P7773P66afzySefAPDOO+/w4osvsnDhQqZNm7bLnGwjRozggQceYP78+dxzzz1cfvnlKffjjBkz+OCDD5g7dy4LFixg/vz5zJ49m+nTp9O6dWsWLlzIkiVLGDhwIKNGjaJ169bMnDmTmTNnpuy7PHT6SEQqJTs7m9WrVzNhwoT4mkGRk08+mQ8//JDp06czbdo0unTpwpIl0Y3yUp0+WrFiBW3btuXII48E4IILLuChhx7iqquu4vDDD2fZsmXMnTuXa665htmzZ7Njxw569eoVr180mujatSurV68udhtFiemjjz5i8ODBxY5kZs+ezZ///GcABg8ezA9+8AMA3nzzTU499VQyMjLIyMjglFNOAWDLli289dZbnHXWWXEf27Zt263fZDNmzGDGjBl06dIl7ueDDz6gV69e/OIXv+D6668nLy9vl9eYDkoKInXErFkl1zVuXHp9q1al16cyZMgQrr32WmbNmsXGjRt3qWvRogVnn302Z599Nnl5ecyePZuuXSv3k6TevXszbdo0GjRoQP/+/Rk2bBg7duxgzJgxcZuGDRsC0amhwsLCYvspSkwbNmygZ8+eTJ48mSFDhlQqtp07d9K8efNyXy9xd2688UYuueSS3ereffddpk6dyq9+9StOPPFEbrrppkrFWBqdPhKRShs+fDijR4+mU6dOu5S//vrr8TdkNm/ezKpVq8o8VU27du1YvXo1K1euBODpp5/mJz+JbkPeq1cvxo4dS48ePcjMzGTjxo2sWLGCjh07Vij+Vq1aceedd/Lb3/52t7revXvzxz9Gty+fNm0aX3/9NQA9e/bkL3/5CwUFBWzZsoUpU6KJoPfff3/atm3Ln/70JyA62C9cmPrOwCeffDLjxo1jy5YtAHz22WesW7eOtWvX0rhxY84991yuu+463n33XQD2228/Nm/eXKHXWxqNFESk0tq0aVPsVzjnz5/PFVdcQf369dm5cycXX3wx3bp1Y/Xq1fGpmyLDhw/fpY+MjAyeeOIJzjrrLAoLC+nWrRuXXhrdjvvYY4/lyy+/pHfv3kB0CuuLL76o1A+3TjvtNG6++ebdrm2MHj2a/Px8OnTowPHHHx8ntW7dujFkyBCys7M54IAD6NSpE82aNQPg2Wef5bLLLuP2229n+/btDB06lM6dO5e6/ZNOOolly5bRo0cPAJo2bcozzzzDypUrue6669hnn31o0KABjzzyCBBdtxg4cGB8baGq1Lp7NOfm5rpusiMCy5Yt4+ijj67pMPZqW7ZsoWnTpmzdupXevXvz6KOPcswxx9R0WMW+N8xsvrun/J6tRgoiIhU0YsQIli5dSkFBARdccMEekRAqS0lBRKSCiq411CW60CwiIjElBRERiSkpiIhITElBRERiSgoiUmHpnjp7ypQpdOnSJV7/97//PQA333xzPPlcSbKysujUqRPZ2dn85Cc/4eOPP47rjj/++GLXGTZsGC+88EL5dkKSxYsXx6+pRYsWtG3blpycHPr371/mPiZNmsTSpUsrFUdF6dtHIlJh6Zw6e/v27YwYMYK5c+fSpk0btm3bVuIcRiUpmpJ69OjR3H777Tz22GMAvPXWW+Xqpzw6deoUv65hw4aRl5fHmWeeWa4+Jk2aRF5eHu3bt09DhKXTSEFEKiVdU2dv3ryZwsJCWrZsCURzGbVr165CMfbo0SOeFhuiXwtDNAXFFVdcQbt27ejfvz/r1q2L20ydOpWjjjqKrl27MmrUqHj08+233zJ8+HC6d+9Oly5dePnll8sUw4wZM+jRowfHHHMMZ511VjydxQ033ED79u3Jzs7m2muv5a233mLy5Mlcd9115OTksGrVqgq95orSSEGkjugzvs9uZf/R4T+4vNvlbN2+lUHPDtqtfljOMIblDGPD1g2c+fyun2ZnDZtVpu0OHTqUW2+9lby8PBYtWsTw4cPjqSJGjhzJz3/+cx588EH69+/PhRdeSOvWrQF2m+bigQce2GUG0BYtWjBkyBAOPfRQTjzxRPLy8sjPz2effcr/WXb69Omcdtppu5W/9NJLrFixgqVLl/Lll1/Svn17hg8fTkFBAZdccgmzZ8+mbdu2uyS6O+64g379+jFu3Di++eYbunfvTv/+/WnSpEmJ29+wYQO33347r776Kk2aNOGuu+7i3nvvZeTIkbz00kssX74cM+Obb76hefPmDBkypEIjjKqgpCAilZKuqbMBHn/8cRYvXsyrr77KPffcwyuvvML4optAlEHfvn356quvaNq0Kbfddttu9bNnzyY/P5969erRunVr+vXrB8Dy5cs57LDDaNu2LQD5+fk8+uijQPSJf/LkyfE1jYKCAj755JNSpxx5++23Wbp0KT179gTgu+++o0ePHjRr1oyMjAwuuugi8vLy4tFITVJSEKkjSvtk37hB41LrWzVuVeaRQXHSOXV2p06d6NSpE+eddx5t27YtV1KYOXMmzZs355xzzmH06NHce++9ZV63JO7Oiy++WK5TWe7OgAEDmDBhwm51c+fO5bXXXuOFF17gwQcf5PXXX690jJWhawoiUmnpmDp7y5YtzEq4ycOCBQs49NBDyx1b/fr1GTt2LE899RRfffXVLnW9e/dm4sSJ7Nixg88//zyebbRdu3Z8+OGH8YXtiRMnxuucfPLJPPDAA/FtOd97772UMRx33HG8+eab8TTg3377Lf/85z/ZsmULmzZtYtCgQdx3333xFNvpmha7LJQURKTSSps6Ozc3l+zsbHr06BFPnQ27347z/vvv32Vdd+fuu++mXbt25OTkMHr06F1GCbfffjtt2rSJH6U58MADyc/P56GHHtql/PTTT+eII46gffv2nH/++fG01Y0aNeLhhx+O77O83377xdNi//rXv2b79u1kZ2fToUMHfv3rX6fcP5mZmYwfP578/Px4XyxfvpzNmzeTl5dHdnY2J5xwQjySGTp0KGPGjKFLly7VfqFZU2eL1FKaOju9iqbFdndGjhzJEUccwdVXX13TYZVJZabO1khBRKQYjz32GDk5OXTo0IFNmzYVe5vMukgXmkVEinH11VfXmpFBVdJIQaQWq22nfyX9KvueUFIQqaUyMjLYuHGjEoPE3J2NGzeSkZFR4T50+kiklmrTpg1r1qxh/fr1NR2K7EEyMjJSfhurNEoKIrVUgwYN4l/cilQVnT4SEZGYkoKIiMSUFEREJKakICIiMSUFERGJKSmIiEgs7UnBzOqZ2XtmNqWYuoZmNtHMVprZHDPLSnc8IiJSsuoYKVwJLCuh7iLga3f/MXAfcFc1xCMiIiVIa1IwszbAYODxEpqcCjwZll8ATjQzS2dMIiJSsnSPFMYC/w3sLKH+IOBTAHcvBDYBLZMbmdkIM5tnZvP0k34RkfRJW1IwszxgnbvPr2xf7v6ou+e6e25mZmYVRCciIsVJ50ihJzDEzFYDzwH9zOyZpDafAQcDmFl9oBmwERERqRFpSwrufqO7t3H3LGAo8Lq7n5vUbDJwQVg+M7TRPMAiIjWk2mdJNbNbgXnuPhn4A/C0ma0EviJKHiIiUkOqJSm4+yxgVli+KaG8ADirOmIQEZHU9ItmERGJKSmIiEhMSUFERGJKCiIiElNSEBGRmJKCiIjElBRERCSmpCAiIjElBRERiSkpiIhITElBRERiSgoiIhJTUhARkZiSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISExJQUREYkoKIiISU1IQEZGYkoKIiMSUFEREJKakICIiMSUFERGJKSmIiEhMSUFERGJKCiIiElNSEBGRmJKCiIjElBRERCSmpCAiIjElBRERiSkpiIhILG1JwcwyzGyumS00s/fN7JZi2gwzs/VmtiA8Lk5XPCIiklr9NPa9Dejn7lvMrAHwDzOb5u5vJ7Wb6O5XpDEOEREpo7QlBXd3YEt42iA8PF3bExGRykvrNQUzq2dmC4B1wCvuPqeYZj8zs0Vm9oKZHVxCPyPMbJ6ZzVu/fn06QxYR2aulNSm4+w53zwHaAN3NrGNSk78AWe6eDbwCPFlCP4+6e66752ZmZqYzZBGRvVq1fPvI3b8BZgIDk8o3uvu28PRxoGt1xCMiIsVL57ePMs2seVhuBAwAlie1OTDh6RBgWbriERGR1NL57aMDgSfNrB5R8nne3aeY2a3APHefDIwysyFAIfAVMCyN8YiISAoWfUmo9sjNzfV58+bVdBgiIrWKmc1399xU7fSLZhERiSkpiIhITElBRERipSYFMzs3YblnUp2mphARqWNSjRSuSVh+IKlueBXHIiIiNSxVUrASlot7LiIitVyqpOAlLBf3XEREarlUP147yswWEY0KDg/LhOeHpTUyERGpdqmSwtHVEoWIiOwRSk0K7v5x4nMzawn0Bj5x9/npDExERKpfqq+kTima7jpMXreE6FtHT5vZVekPT0REqlOqC81t3X1JWL6Q6EY5pwDHoq+kiojUOamSwvaE5ROBqQDuvhnYma6gRESkZqS60Pypmf0XsAY4BpgO8f0RGqQ5NhERqWapRgoXAR2I7nPw83AHNYDjgCfSF5aIiNSEVN8+WgdcWkz5TKLba4qISB1SalIws8ml1bv7kKoNR0REalKqawo9gE+BCcAcNN+RiEidliop/AgYAOQDZwN/BSa4+/vpDkxERKpfqRea3X2Hu0939wuILi6vBGbpXgoiInVTqpECZtYQGEw0WsgC7gdeSm9YIiJSE1JdaH4K6Ej0o7VbEn7dLCIidVCqkcK5wLfAlcAos/g6swHu7vunMTYREalmqX6nkOrHbSIiUofooC8iIjElBRERiSkpiIhITElBRERiSgoiIhJTUhARkZiSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISCxtScHMMsxsrpktNLP3zeyWYto0NLOJZrbSzOaYWVa64hERkdTSOVLYBvRz985ADjDQzI5LanMR8LW7/xi4D7grjfGIiEgKaUsKHtkSnjYID09qdirwZFh+ATjREubnFhGR6pXWawpmVs/MFgDrgFfcfU5Sk4OATwHcvRDYBLQspp8RZjbPzOatX78+nSGLiOzV0poUwj2ec4A2QHcz61jBfh5191x3z83MzKzSGEVE5HvV8u0jd/8GmAkMTKr6DDgYwMzqA82AjdURk4iI7C6d3z7KNLPmYbkRMABYntRsMnBBWD4TeN3dk687iIhINUl1j+bKOBB40szqESWf5919ipndCsxz98nAH4CnzWwl8BUwNI3xiIhICmlLCu6+COhSTPlNCcsFwFnpikFERMpHv2gWEZGYkoKIiMSUFEREJKakICIiMSUFERGJKSmIiEhMSUFERGJKCiIiElNSEBGRmJKCiIjElBRERCSmpCAiIjElBRERiSkpiIhITElBRERiSgoiIhJTUhARkZiSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISExJQUREYkoKIiISU1IQEZGYkoKIiMSUFEREJKakICIiMSUFERGJKSmIiEhMSUFERGJKCiIiElNSEBGRmJKCiIjElBRERCSWtqRgZgeb2UwzW2pm75vZlcW06WNmm8xsQXjclK54REQktfpp7LsQ+IW7v2tm+wHzzewVd1+a1O4Nd89LYxwiIlJGaRspuPvn7v5uWN4MLAMOStf2RESk8qrlmoKZZQFdgDnFVPcws4VmNs3MOpSw/ggzm2dm89avX5/OUEVE9mppTwpm1hR4EbjK3f+VVP0ucKi7dwYeACYV14e7P+ruue6em5mZmdZ4RUT2ZmlNCmbWgCghPOvuf06ud/d/ufuWsDwVaGBmrdIZk4iIlCyd3z4y4A/AMne/t4Q2PwrtMLPuIZ6N6YpJRERKl85vH/UEzgMWm9mCUPb/gEMA3P13wJnAZWZWCPwbGOrunsaYRESkFGlLCu7+D8BStHkQeDBdMYiISPnoF80iIhJTUhARkZiSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISExJQUREYkoKIiISU1IQEZGYkoKIiMSUFEREJKakICIiMSUFERGJKSmIiEhMSUFERGJKCiIiElNSEBGRmJKCiIjElBRERCSmpCAiIjElBRERiSkpiIhITElBRERiSgoiIhJTUhARkZiSgoiIxJQUREQkpqQgIiIxJQUREYkpKYiISExJQUREYubuNR1DuZjZeuDjCq7eCthQheFUlT01LthzY1Nc5aO4yqcuxnWou2emalTrkkJlmNk8d8+t6TiS7alxwZ4bm+IqH8VVPntzXDp9JCIiMSUFERGJ7W1J4dGaDqAEe2pcsOfGprjKR3GVz14b1151TUFEREq3t40URESkFEoKIiISqxNJwczGmNlyM1tkZi+ZWfMS2o0zs3VmtiSpvIWZvWJmH4R/fxDKzczuN7OVoe9j0hTXQDNbEbZzQ0L5G2a2IDzWmtmkUN7HzDYl1N1UzXGNN7OPErafE8pren89G8qXhL91g1Be0/urrZnNCeUTzWzfUN4wPF8Z6rPKGddZZva+me00sxK/pmhmV4Z98r6ZXZVQPjFhn6w2swWhPMvM/p1Q97tqjutmM/ssYfuDEupuDPtrhZmdXM1xFfs+2AP2V9Uev9y91j+Ak4D6Yfku4K4S2vUGjgGWJJXfDdwQlm8oWh8YBEwDDDgOmFPVcQH1gFXAYcC+wEKgfTHtXgTOD8t9gCnp3F+lxQWMB84sZp0a3V9h+xYeE4DL9pD99TwwNCz/LiGuy4HfheWhwMRyxnU00A6YBeSW0KYjsARoDNQHXgV+XEy7/wFuCstZyf9HqjMu4Gbg2mLWaR/2a0Ogbdjf9aoxrmLfB3vA/qrS41edGCm4+wx3LwxP3wbalNBuNvBVMVWnAk+G5SeB0xLKn/LI20BzMzuwiuPqDqx09w/d/TvgubDdmJntD/QDJpV129URVzFqdH+5+9SwbQfmlrB+uVUmLjMzor/dC6Fd8vur6H33AnBiaF/WuJa5+4oUzY4mOhhsDa/h78AZiQ3CNv+DKJFWWlXFVYxTgefcfZu7fwSsJNrv1RJXWY8z5VUF+6tKj191IikkGU6UHcvjAHf/PCx/ARwQlg8CPk1otyaUVWVcZdnGacBr7v6vhLIeZrbQzKaZWYcKxlSZuO4IQ9L7zKxhGdepjrgIp43OA6YnFNfU/moJfJNwMEmMN14n1G8K7avSEqCXmbU0s8ZEnx4PTmrTC/jS3T9IKGtrZu+Z2d/NrFcVx1SWuK4I769xRadDqNr3V0XjKpL8PqjJ/VWlx6/6VRNv+pnZq8CPiqn6pbu/HNr8EigEnq3odtzdzazM39OtprjygccTnr9LNI/JlnC+dRJwRDXGdSPRm29fou9NXw/cWpYVq2l/PQzMdvc3wvOa3l8VVpa4SuPuy8zsLmAG8C2wANiR1CyfXUcJnwOHuPtGM+sKTDKzDokfStIc1yPAbYCHf/+H6CCcUnXsr2LeBzW9vxLblev4VZxakxTcvX9p9WY2DMgDTgynD8rjSzM70N0/D8OrdaH8M3b9lNAmlFVlXKVuw8xaEQ2RT0/Y5r8Slqea2cNm1srdNySUpy2uhE8l28zsCeDasryWdMcV1h8NZAKXJGyzJvfXRqJhe/0wGkiMt2idNWZWH2gW2sdSxVUW7v4H4A/hdfyG6BNj0euqT3QaomtC+23AtrA838xWAUcC86ojLnf/MiG+x4Ap4Wml31+ViSs8H0bS+6Cm9xeVOH6VtKFa/wAGAkuBzDK0zWL3C81j2PVCzd1heTC7XqiZW9VxESXmD4kunBVdoOyQUH8p8GTSOj/i+x8edgc+KXpeHXEBB4Z/DRgL3Lkn7C/gYuAtoNEetr/+xK4Xmi8PyyPZ9ULz8xV8/8+ihAuUof6H4d9DgOVA86TX9vek9pmEC7hEF84/A1pUV1xF76+wfDXRdQSADux6oflDynGhuQriKvZ9sAfsryo9fpX7DbgnPoguOH1KNKRakPAfrTUwNaHdBKKh3naiLHtRKG8JvAZ8QHRVv0UoN+Ahom85LC7tD1bJuAYB/wzb+WUxb5SBSWVXAO+H/yBvA8dXZ1zA62F/LAGeAZruCfuLaEi/KmH9om/T1PT+OozowvdKogTRMJRnhOcrQ/1h5Yzr9PA+3gZ8CfythLjeIDqYLST6hJvYx3jg0qSyn4X9tYDo1Nsp1RkX8HR4/ywCJrNrkvhl2L8rgJ9Wc1wlvQ9qen9V6fFL01yIiEisLn77SEREKkhJQUREYkoKIiISU1IQEZGYkoKIiMSUFKRWM7MtlVj3ijCDpIcfCRaVlzi7pJkdaGZTwnKfouXKMrNZVsoMmQntVifGWkKbVxOmhhApFyUF2Zu9CfQHPk4q/ynRNBhHACOIpl0ocg3wWLVEV3FPE83AKlJuSgpSJ4RP92Msmm9+sZn9PJTvE6a1WG7RXPNTzexMAHd/z91XF9NdabNL/oxdJ9or2n53M/u/MCnaW2bWLpQPM7NJYdurw+jkmtDubTNrkdDNeRbNx7/EzLqH9Vua2QyL5tB/nOgHSUXbnGRm80PdiIR+JhPNZyRSbkoKUlecAeQAnYk+/Y8JB/IziKY2aU80c2qPMvRV7OySZtYW+NqjuW6SLQd6uXsX4CbgNwl1HUMc3YA7gK2h3f8B5ye0a+zuOUSf8seFstHAP9y9A/AS0RQHRYa7e1cgFxhlZi0B3P1roGHRc5HyqDUT4omkcAIwwd13EE0Q9neig/AJwJ/cfSfwhZnNrMQ2DgTWl1DXDHjSzI4gmt2zQULdTHffDGw2s03AX0L5YiA7od0EiO77YWb7W3Rnr958P5//X83s64T2o8ysaKLEg4lOdxVNqLeOaJqEXSbYE0lFIwWR3ZU0u+S/ieYrKs5tRAf/jsApSe0SRxY7E57vZNcPZslzzpQ4B42Z9SEaEfVw987Ae0nbzAjxipSLkoLUFW8APzezemaWSfQJey7RxeSfhWsLBxDdmjOVycD54TrFccAmj6YL/yfRqajiNOP7aYmHVfA1FF0HOSFscxMwGzg7lP8UKPpWUTOiU1lbzewoolkwCe2MaGbY1RWMQ/ZiSgpSV7xENKvmQqJZXP/b3b8gurf1GqLZJZ8hmsVyE4CZjTKzNUQjgUXhQi7AVKJpmVcSfdPocgB3/xZYZWY/Lmb7dwO/NbP3qPhp2YKw/u+Ai0LZLUBvM3uf6DTSJ6F8OlDfzJYBdxLN/lqkK/C2f3+3N5Ey0yypUueZWVOP7rrWkmj00DMkjIr0dTrQ1d1/VaVBViEz+19gsru/VtOxSO2jC82yN5gSLtruC9xW0YQA4O4v1YJv9SxRQpCK0khBRERiuqYgIiIxJQUREYkpKYiISExJQUREYkoKIiIS+//pVaNwjKaKmwAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
    " - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "execution_count": 10, + "id": "3bac9cff", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", @@ -2522,11 +2803,11 @@ "p = Maxpolydegree-1\n", "I = np.eye(p,p)\n", "# Decide which values of lambda to use\n", - "nlambdas = 1\n", + "nlambdas = 6\n", "MSEOwnRidgePredict = np.zeros(nlambdas)\n", "MSERidgePredict = np.zeros(nlambdas)\n", "\n", - "lambdas = np.logspace(-2, 2, nlambdas)\n", + "lambdas = np.logspace(-4, 2, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", " OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n", @@ -2564,8 +2845,10 @@ }, { "cell_type": "markdown", - "id": "9cd41fb5", - "metadata": {}, + "id": "9431aefa", + "metadata": { + "editable": true + }, "source": [ "We see here, when compared to the code which includes explicitely the\n", "intercept column, that our MSE value is actually smaller. This is\n", @@ -2579,16 +2862,20 @@ }, { "cell_type": "markdown", - "id": "85a6e4f5", - "metadata": {}, + "id": "a06f0a20", + "metadata": { + "editable": true + }, "source": [ "## Friday September 9" ] }, { "cell_type": "markdown", - "id": "a2a65ad6", - "metadata": {}, + "id": "491bae26", + "metadata": { + "editable": true + }, "source": [ "## Linking the regression analysis with a statistical interpretation\n", "\n", @@ -2614,8 +2901,10 @@ }, { "cell_type": "markdown", - "id": "e8d65034", - "metadata": {}, + "id": "a2c774e7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*} \n", @@ -2628,8 +2917,10 @@ }, { "cell_type": "markdown", - "id": "6fa8ce61", - "metadata": {}, + "id": "fdf93799", + "metadata": { + "editable": true + }, "source": [ "The randomness of $\\varepsilon_i$ implies that\n", "$\\mathbf{y}_i$ is also a random variable. In particular,\n", @@ -2645,8 +2936,10 @@ }, { "cell_type": "markdown", - "id": "e503e716", - "metadata": {}, + "id": "61783020", + "metadata": { + "editable": true + }, "source": [ "## Assumptions made\n", "\n", @@ -2657,8 +2950,10 @@ }, { "cell_type": "markdown", - "id": "6590f6c6", - "metadata": {}, + "id": "579a3e8e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", @@ -2667,8 +2962,10 @@ }, { "cell_type": "markdown", - "id": "36c66ae3", - "metadata": {}, + "id": "8919ec8d", + "metadata": { + "editable": true + }, "source": [ "We approximate this function with our model from the solution of the linear regression equations, that is our\n", "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with" @@ -2676,8 +2973,10 @@ }, { "cell_type": "markdown", - "id": "c794166a", - "metadata": {}, + "id": "8e6d1273", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", @@ -2686,8 +2985,10 @@ }, { "cell_type": "markdown", - "id": "23801ca2", - "metadata": {}, + "id": "9b397651", + "metadata": { + "editable": true + }, "source": [ "## Expectation value and variance\n", "\n", @@ -2696,8 +2997,10 @@ }, { "cell_type": "markdown", - "id": "859a80be", - "metadata": {}, + "id": "b87296ba", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*} \n", @@ -2710,8 +3013,10 @@ }, { "cell_type": "markdown", - "id": "8df27f31", - "metadata": {}, + "id": "075a29e0", + "metadata": { + "editable": true + }, "source": [ "while\n", "its variance is" @@ -2719,8 +3024,10 @@ }, { "cell_type": "markdown", - "id": "efab0a0d", - "metadata": {}, + "id": "ab22e3a9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n", @@ -2740,8 +3047,10 @@ }, { "cell_type": "markdown", - "id": "c41b6e91", - "metadata": {}, + "id": "8064c896", + "metadata": { + "editable": true + }, "source": [ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)." @@ -2749,8 +3058,10 @@ }, { "cell_type": "markdown", - "id": "2f7a7871", - "metadata": {}, + "id": "9ecea4d2", + "metadata": { + "editable": true + }, "source": [ "## Expectation value and variance for $\\boldsymbol{\\beta}$\n", "\n", @@ -2759,8 +3070,10 @@ }, { "cell_type": "markdown", - "id": "0008f742", - "metadata": {}, + "id": "ddbbbf7e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n", @@ -2769,8 +3082,10 @@ }, { "cell_type": "markdown", - "id": "72111c95", - "metadata": {}, + "id": "2d2e304c", + "metadata": { + "editable": true + }, "source": [ "This means that the estimator of the regression parameters is unbiased.\n", "\n", @@ -2781,8 +3096,10 @@ }, { "cell_type": "markdown", - "id": "52790c16", - "metadata": {}, + "id": "8902a5b2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{eqnarray*}\n", @@ -2810,8 +3127,10 @@ }, { "cell_type": "markdown", - "id": "864688c4", - "metadata": {}, + "id": "33a2838d", + "metadata": { + "editable": true + }, "source": [ "where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n", "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n", @@ -2830,8 +3149,10 @@ }, { "cell_type": "markdown", - "id": "37738b35", - "metadata": {}, + "id": "85db379b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", @@ -2840,8 +3161,10 @@ }, { "cell_type": "markdown", - "id": "aeff32a4", - "metadata": {}, + "id": "18c22f28", + "metadata": { + "editable": true + }, "source": [ "We see clearly that \n", "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", @@ -2851,8 +3174,10 @@ }, { "cell_type": "markdown", - "id": "3321cbc4", - "metadata": {}, + "id": "8bc69031", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", @@ -2861,8 +3186,10 @@ }, { "cell_type": "markdown", - "id": "486213d7", - "metadata": {}, + "id": "d6544f85", + "metadata": { + "editable": true + }, "source": [ "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n", "\n", @@ -2871,8 +3198,10 @@ }, { "cell_type": "markdown", - "id": "ca94083c", - "metadata": {}, + "id": "2457b453", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", @@ -2881,8 +3210,10 @@ }, { "cell_type": "markdown", - "id": "167df200", - "metadata": {}, + "id": "b051314b", + "metadata": { + "editable": true + }, "source": [ "The difference is non-negative definite since each component of the\n", "matrix product is non-negative definite. \n", @@ -2891,8 +3222,10 @@ }, { "cell_type": "markdown", - "id": "b8c4a1e4", - "metadata": {}, + "id": "cfda23b3", + "metadata": { + "editable": true + }, "source": [ "## Deriving OLS from a probability distribution\n", "\n", @@ -2912,8 +3245,10 @@ }, { "cell_type": "markdown", - "id": "148b3cdb", - "metadata": {}, + "id": "a23408ee", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -2922,8 +3257,10 @@ }, { "cell_type": "markdown", - "id": "686d7fa3", - "metadata": {}, + "id": "202e37fb", + "metadata": { + "editable": true + }, "source": [ "## Independent and Identically Distrubuted (iid)\n", "\n", @@ -2933,8 +3270,10 @@ }, { "cell_type": "markdown", - "id": "17caa70f", - "metadata": {}, + "id": "dbcf24e6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n", @@ -2943,8 +3282,10 @@ }, { "cell_type": "markdown", - "id": "011cc0f0", - "metadata": {}, + "id": "06ceed52", + "metadata": { + "editable": true + }, "source": [ "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n", "\n", @@ -2953,8 +3294,10 @@ }, { "cell_type": "markdown", - "id": "fd47caec", - "metadata": {}, + "id": "a7fb86c6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n", @@ -2963,8 +3306,10 @@ }, { "cell_type": "markdown", - "id": "be92a8b7", - "metadata": {}, + "id": "05f05891", + "metadata": { + "editable": true + }, "source": [ "We will write this in a more compact form reserving $\\boldsymbol{D}$ for the domain of events, including the ouputs (targets) and the inputs. That is\n", "in case we have a simple one-dimensional input and output case" @@ -2972,8 +3317,10 @@ }, { "cell_type": "markdown", - "id": "7af51d2b", - "metadata": {}, + "id": "b94d1190", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})].\n", @@ -2982,8 +3329,10 @@ }, { "cell_type": "markdown", - "id": "856dff7d", - "metadata": {}, + "id": "d4c0ad8e", + "metadata": { + "editable": true + }, "source": [ "In the more general case the various inputs should be replaced by the possible features represented by the input data set $\\boldsymbol{X}$. \n", "We can now rewrite the above probability as" @@ -2991,8 +3340,10 @@ }, { "cell_type": "markdown", - "id": "5a446379", - "metadata": {}, + "id": "65a46e42", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -3001,16 +3352,20 @@ }, { "cell_type": "markdown", - "id": "4204c439", - "metadata": {}, + "id": "679d99d2", + "metadata": { + "editable": true + }, "source": [ "It is a conditional probability (see below) and reads as the likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters $\\boldsymbol{\\beta}$." ] }, { "cell_type": "markdown", - "id": "d3bc6574", - "metadata": {}, + "id": "d4362fd5", + "metadata": { + "editable": true + }, "source": [ "## Maximum Likelihood Estimation (MLE)\n", "\n", @@ -3038,8 +3393,10 @@ }, { "cell_type": "markdown", - "id": "5605b443", - "metadata": {}, + "id": "9e242cda", + "metadata": { + "editable": true + }, "source": [ "## A new Cost Function\n", "\n", @@ -3048,8 +3405,10 @@ }, { "cell_type": "markdown", - "id": "7eb23f48", - "metadata": {}, + "id": "a7543db7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n", @@ -3058,16 +3417,20 @@ }, { "cell_type": "markdown", - "id": "6167023f", - "metadata": {}, + "id": "fcc3cb78", + "metadata": { + "editable": true + }, "source": [ "which becomes" ] }, { "cell_type": "markdown", - "id": "a4443fe7", - "metadata": {}, + "id": "f76e3a84", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n", @@ -3076,16 +3439,20 @@ }, { "cell_type": "markdown", - "id": "358b1b20", - "metadata": {}, + "id": "aaacd43a", + "metadata": { + "editable": true + }, "source": [ "Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely" ] }, { "cell_type": "markdown", - "id": "50135271", - "metadata": {}, + "id": "44194843", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n", @@ -3094,16 +3461,20 @@ }, { "cell_type": "markdown", - "id": "4f04c33f", - "metadata": {}, + "id": "a1203a00", + "metadata": { + "editable": true + }, "source": [ "which leads to the well-known OLS equation for the optimal paramters $\\beta$" ] }, { "cell_type": "markdown", - "id": "9adef013", - "metadata": {}, + "id": "cfb9ea5c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", @@ -3112,16 +3483,20 @@ }, { "cell_type": "markdown", - "id": "88521aab", - "metadata": {}, + "id": "5eb9a0ea", + "metadata": { + "editable": true + }, "source": [ "Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics." ] }, { "cell_type": "markdown", - "id": "48e28fb9", - "metadata": {}, + "id": "147ba03e", + "metadata": { + "editable": true + }, "source": [ "## More basic Statistics and Bayes' theorem\n", "\n", @@ -3138,8 +3513,10 @@ }, { "cell_type": "markdown", - "id": "d75b9c96", - "metadata": {}, + "id": "15b862d7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X \\cup Y)= p(X)+p(Y)-p(X \\cap Y).\n", @@ -3148,16 +3525,20 @@ }, { "cell_type": "markdown", - "id": "597adf34", - "metadata": {}, + "id": "c0ed74ec", + "metadata": { + "editable": true + }, "source": [ "**The product rule (aka joint probability) is given by.**" ] }, { "cell_type": "markdown", - "id": "ae73334c", - "metadata": {}, + "id": "1e5c85a3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X \\cup Y)= p(X,Y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(X),\n", @@ -3166,8 +3547,10 @@ }, { "cell_type": "markdown", - "id": "6b91371c", - "metadata": {}, + "id": "eb1ad707", + "metadata": { + "editable": true + }, "source": [ "where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n", "\n", @@ -3176,8 +3559,10 @@ }, { "cell_type": "markdown", - "id": "60c017df", - "metadata": {}, + "id": "f5862c9a", + "metadata": { + "editable": true + }, "source": [ "## Marginal Probability\n", "\n", @@ -3186,8 +3571,10 @@ }, { "cell_type": "markdown", - "id": "bfc5883c", - "metadata": {}, + "id": "a93994dd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X)=\\sum_{i=0}^{n-1}p(X,Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert Y=y_i)p(Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert y_i)p(y_i).\n", @@ -3196,8 +3583,10 @@ }, { "cell_type": "markdown", - "id": "c5227ac5", - "metadata": {}, + "id": "48362ea4", + "metadata": { + "editable": true + }, "source": [ "## Conditional Probability\n", "\n", @@ -3206,8 +3595,10 @@ }, { "cell_type": "markdown", - "id": "6e13d49f", - "metadata": {}, + "id": "a736264f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)}=\\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}.\n", @@ -3216,8 +3607,10 @@ }, { "cell_type": "markdown", - "id": "9f576b4b", - "metadata": {}, + "id": "2ca55904", + "metadata": { + "editable": true + }, "source": [ "## Bayes' Theorem\n", "\n", @@ -3226,8 +3619,10 @@ }, { "cell_type": "markdown", - "id": "79f8fe23", - "metadata": {}, + "id": "3232736a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)},\n", @@ -3236,16 +3631,20 @@ }, { "cell_type": "markdown", - "id": "b5b3d792", - "metadata": {}, + "id": "a05c0eb9", + "metadata": { + "editable": true + }, "source": [ "which we can rewrite as" ] }, { "cell_type": "markdown", - "id": "1fff2eb5", - "metadata": {}, + "id": "2d8a77dc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}=\\frac{p(Y\\vert X)p(X)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)},\n", @@ -3254,16 +3653,20 @@ }, { "cell_type": "markdown", - "id": "ebfbf1a4", - "metadata": {}, + "id": "c2983edd", + "metadata": { + "editable": true + }, "source": [ "which is Bayes' theorem. It allows us to evaluate the uncertainty in in $X$ after we have observed $Y$. We can easily interchange $X$ with $Y$." ] }, { "cell_type": "markdown", - "id": "99f96e4d", - "metadata": {}, + "id": "285d34f1", + "metadata": { + "editable": true + }, "source": [ "## Interpretations of Bayes' Theorem\n", "\n", @@ -3279,8 +3682,10 @@ }, { "cell_type": "markdown", - "id": "46017fbc", - "metadata": {}, + "id": "4daab924", + "metadata": { + "editable": true + }, "source": [ "## Example of Usage of Bayes' theorem\n", "\n", @@ -3298,8 +3703,10 @@ }, { "cell_type": "markdown", - "id": "45acf8d6", - "metadata": {}, + "id": "134a7757", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X=1\\vert Y=1) =0.8.\n", @@ -3308,8 +3715,10 @@ }, { "cell_type": "markdown", - "id": "d7189c6c", - "metadata": {}, + "id": "c43dde74", + "metadata": { + "editable": true + }, "source": [ "This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of $80\\%$ for having cancer.\n", "It is however not correct, as the following Bayesian analysis shows." @@ -3317,8 +3726,10 @@ }, { "cell_type": "markdown", - "id": "095f1344", - "metadata": {}, + "id": "18a5f41d", + "metadata": { + "editable": true + }, "source": [ "## Doing it correctly\n", "\n", @@ -3328,8 +3739,10 @@ }, { "cell_type": "markdown", - "id": "7a421514", - "metadata": {}, + "id": "31edd0a3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(Y=1) =0.004.\n", @@ -3338,16 +3751,20 @@ }, { "cell_type": "markdown", - "id": "f6cb9991", - "metadata": {}, + "id": "f9171778", + "metadata": { + "editable": true + }, "source": [ "We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have" ] }, { "cell_type": "markdown", - "id": "4ca65a78", - "metadata": {}, + "id": "33f83b25", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X=1\\vert Y=0) =0.1.\n", @@ -3356,16 +3773,20 @@ }, { "cell_type": "markdown", - "id": "940490f2", - "metadata": {}, + "id": "b23ddfdd", + "metadata": { + "editable": true + }, "source": [ "Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute" ] }, { "cell_type": "markdown", - "id": "fd376a0c", - "metadata": {}, + "id": "0c52e6db", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(Y=1\\vert X=1)=\\frac{p(X=1\\vert Y=1)p(Y=1)}{p(X=1\\vert Y=1)p(Y=1)+p(X=1\\vert Y=0)p(Y=0)}=\\frac{0.8\\times 0.004}{0.8\\times 0.004+0.1\\times 0.996}=0.031.\n", @@ -3374,16 +3795,20 @@ }, { "cell_type": "markdown", - "id": "891b9eda", - "metadata": {}, + "id": "bcbb9fb7", + "metadata": { + "editable": true + }, "source": [ "That is, in case of a positive test, there is only a $3\\%$ chance of having breast cancer!" ] }, { "cell_type": "markdown", - "id": "3734a062", - "metadata": {}, + "id": "5d5effd4", + "metadata": { + "editable": true + }, "source": [ "## Bayes' Theorem and Ridge and Lasso Regression\n", "\n", @@ -3397,8 +3822,10 @@ }, { "cell_type": "markdown", - "id": "2b6262c1", - "metadata": {}, + "id": "12191b9e", + "metadata": { + "editable": true + }, "source": [ "## Test Function for what happens with OLS, Ridge and Lasso\n", "\n", @@ -3414,8 +3841,11 @@ { "cell_type": "code", "execution_count": 11, - "id": "6028f12f", - "metadata": {}, + "id": "80b8d381", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -3485,16 +3915,20 @@ }, { "cell_type": "markdown", - "id": "1e94a8e4", - "metadata": {}, + "id": "3d846a7b", + "metadata": { + "editable": true + }, "source": [ "How can we understand this?" ] }, { "cell_type": "markdown", - "id": "e89f9a23", - "metadata": {}, + "id": "cdba3cee", + "metadata": { + "editable": true + }, "source": [ "## Invoking Bayes' theorem\n", "\n", @@ -3505,8 +3939,10 @@ }, { "cell_type": "markdown", - "id": "bcc23e22", - "metadata": {}, + "id": "f2f7790a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})],\n", @@ -3515,16 +3951,20 @@ }, { "cell_type": "markdown", - "id": "ab35b334", - "metadata": {}, + "id": "14655685", + "metadata": { + "editable": true + }, "source": [ "is given by" ] }, { "cell_type": "markdown", - "id": "7bad9dac", - "metadata": {}, + "id": "a87a0d7d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -3533,16 +3973,20 @@ }, { "cell_type": "markdown", - "id": "d58ce44e", - "metadata": {}, + "id": "c49cf8f2", + "metadata": { + "editable": true + }, "source": [ "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$? That is, how can we define the posterior probability" ] }, { "cell_type": "markdown", - "id": "e302a211", - "metadata": {}, + "id": "093333d4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n", @@ -3551,16 +3995,20 @@ }, { "cell_type": "markdown", - "id": "ffc980d1", - "metadata": {}, + "id": "1b9fa92a", + "metadata": { + "editable": true + }, "source": [ "Bayes' theorem comes to our rescue here since (omitting the normalization constant)" ] }, { "cell_type": "markdown", - "id": "860cdf65", - "metadata": {}, + "id": "53c11861", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n", @@ -3569,16 +4017,20 @@ }, { "cell_type": "markdown", - "id": "8ad984c0", - "metadata": {}, + "id": "2f8298c1", + "metadata": { + "editable": true + }, "source": [ "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!" ] }, { "cell_type": "markdown", - "id": "d0bee78c", - "metadata": {}, + "id": "87d74e6a", + "metadata": { + "editable": true + }, "source": [ "## Ridge and Bayes\n", "\n", @@ -3591,8 +4043,10 @@ }, { "cell_type": "markdown", - "id": "bf039e41", - "metadata": {}, + "id": "a049d0ea", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -3601,16 +4055,20 @@ }, { "cell_type": "markdown", - "id": "fffb60aa", - "metadata": {}, + "id": "423dbbec", + "metadata": { + "editable": true + }, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] }, { "cell_type": "markdown", - "id": "67b49eab", - "metadata": {}, + "id": "8f81c0dd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -3619,8 +4077,10 @@ }, { "cell_type": "markdown", - "id": "cae7309e", - "metadata": {}, + "id": "d44ba548", + "metadata": { + "editable": true + }, "source": [ "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n", "did for OLS, this is most conveniently done by taking the negative\n", @@ -3630,8 +4090,10 @@ }, { "cell_type": "markdown", - "id": "e15e70d1", - "metadata": {}, + "id": "759e1a46", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -3640,16 +4102,20 @@ }, { "cell_type": "markdown", - "id": "36d005b5", - "metadata": {}, + "id": "32b1de62", + "metadata": { + "editable": true + }, "source": [ "and replacing $1/2\\tau^2$ with $\\lambda$ we have" ] }, { "cell_type": "markdown", - "id": "4deb0ca9", - "metadata": {}, + "id": "2378e1f6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -3658,16 +4124,20 @@ }, { "cell_type": "markdown", - "id": "84fcc55c", - "metadata": {}, + "id": "787f1ed7", + "metadata": { + "editable": true + }, "source": [ "which is our Ridge cost function! Nice, isn't it?" ] }, { "cell_type": "markdown", - "id": "47d7c3ca", - "metadata": {}, + "id": "401c6c4e", + "metadata": { + "editable": true + }, "source": [ "## Lasso and Bayes\n", "\n", @@ -3676,8 +4146,10 @@ }, { "cell_type": "markdown", - "id": "8e745a19", - "metadata": {}, + "id": "3f7aece4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -3686,16 +4158,20 @@ }, { "cell_type": "markdown", - "id": "55142cc6", - "metadata": {}, + "id": "deb8ed7f", + "metadata": { + "editable": true + }, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] }, { "cell_type": "markdown", - "id": "f627b93a", - "metadata": {}, + "id": "354d2fea", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -3704,8 +4180,10 @@ }, { "cell_type": "markdown", - "id": "1a1bae81", - "metadata": {}, + "id": "834e893c", + "metadata": { + "editable": true + }, "source": [ "Taking the negative\n", "logarithm of the posterior probability and leaving out the\n", @@ -3714,8 +4192,10 @@ }, { "cell_type": "markdown", - "id": "0c8a5a7b", - "metadata": {}, + "id": "9f14f991", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -3724,16 +4204,20 @@ }, { "cell_type": "markdown", - "id": "05bcb4ee", - "metadata": {}, + "id": "e749f01f", + "metadata": { + "editable": true + }, "source": [ "and replacing $1/\\tau$ with $\\lambda$ we have" ] }, { "cell_type": "markdown", - "id": "308e688d", - "metadata": {}, + "id": "06f11820", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -3742,16 +4226,20 @@ }, { "cell_type": "markdown", - "id": "e127394e", - "metadata": {}, + "id": "94c9065c", + "metadata": { + "editable": true + }, "source": [ "which is our Lasso cost function!" ] }, { "cell_type": "markdown", - "id": "ca26f6f7", - "metadata": {}, + "id": "20c64501", + "metadata": { + "editable": true + }, "source": [ "## Exercise 1: mean values and variances in linear regression\n", "\n", @@ -3764,8 +4252,10 @@ }, { "cell_type": "markdown", - "id": "a0f78141", - "metadata": {}, + "id": "bc069f72", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", @@ -3774,8 +4264,10 @@ }, { "cell_type": "markdown", - "id": "a285460c", - "metadata": {}, + "id": "ee99deec", + "metadata": { + "editable": true + }, "source": [ "We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our\n", "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we minimized $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, with" @@ -3783,8 +4275,10 @@ }, { "cell_type": "markdown", - "id": "f5f96083", - "metadata": {}, + "id": "7b02bf27", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", @@ -3793,16 +4287,20 @@ }, { "cell_type": "markdown", - "id": "21e62126", - "metadata": {}, + "id": "fdfd39db", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{X}$ is the so-called design matrix." ] }, { "cell_type": "markdown", - "id": "e5730a62", - "metadata": {}, + "id": "8d268428", + "metadata": { + "editable": true + }, "source": [ "**a)**\n", "Show that the expectation value of $\\boldsymbol{y}$ for a given element $i$" @@ -3810,8 +4308,10 @@ }, { "cell_type": "markdown", - "id": "55b4af74", - "metadata": {}, + "id": "23b6f545", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*} \n", @@ -3822,8 +4322,10 @@ }, { "cell_type": "markdown", - "id": "c4fb720a", - "metadata": {}, + "id": "2db7ea8d", + "metadata": { + "editable": true + }, "source": [ "and that\n", "its variance is" @@ -3831,8 +4333,10 @@ }, { "cell_type": "markdown", - "id": "31a32444", - "metadata": {}, + "id": "ce163d77", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*} \\mbox{Var}(y_i) & = \\sigma^2. \n", @@ -3842,8 +4346,10 @@ }, { "cell_type": "markdown", - "id": "18736ca9", - "metadata": {}, + "id": "bbd3a9ca", + "metadata": { + "editable": true + }, "source": [ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$." @@ -3851,8 +4357,10 @@ }, { "cell_type": "markdown", - "id": "889dcad9", - "metadata": {}, + "id": "11a2440e", + "metadata": { + "editable": true + }, "source": [ "**b)**\n", "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ show that" @@ -3860,8 +4368,10 @@ }, { "cell_type": "markdown", - "id": "65d7114c", - "metadata": {}, + "id": "e94c5e5e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\beta}) = \\boldsymbol{\\beta}.\n", @@ -3870,8 +4380,10 @@ }, { "cell_type": "markdown", - "id": "0246cb71", - "metadata": {}, + "id": "cfaebf91", + "metadata": { + "editable": true + }, "source": [ "**c)**\n", "Show finally that the variance of $\\boldsymbol{\\beta}$ is" @@ -3879,8 +4391,10 @@ }, { "cell_type": "markdown", - "id": "9a239b08", - "metadata": {}, + "id": "2fb5c3f9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{eqnarray*}\n", @@ -3891,8 +4405,10 @@ }, { "cell_type": "markdown", - "id": "9b868ddb", - "metadata": {}, + "id": "8121d786", + "metadata": { + "editable": true + }, "source": [ "## Exercise 2: Adding Ridge and Lasso Regression\n", "\n", @@ -3915,8 +4431,11 @@ { "cell_type": "code", "execution_count": 12, - "id": "fe00295c", - "metadata": {}, + "id": "f91041e7", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = np.random.rand(100)\n", @@ -3925,8 +4444,10 @@ }, { "cell_type": "markdown", - "id": "d59b60b2", - "metadata": {}, + "id": "df8b4c9c", + "metadata": { + "editable": true + }, "source": [ "**a)**\n", "Write your own code for the Ridge method (see chapter 3.4 of Hastie *et al.*, equations (3.43) and (3.44)) and compute the parametrization for different values of $\\lambda$. Study the dependence on $\\lambda$ while also varying the strength of the noise in your expression for $y(x)$." @@ -3934,33 +4455,17 @@ }, { "cell_type": "markdown", - "id": "47dc99e1", - "metadata": {}, + "id": "9acc49d5", + "metadata": { + "editable": true + }, "source": [ "**b)**\n", "Our next step is to study the variance of the parameters $\\beta_1$ and $\\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using **Scikit-Learn** to find the parameters $\\beta$. From your results calculate the variance of these parameters (recall that this is equal to the diagonal elements of the matrix $(\\hat{X}^T\\hat{X})+\\lambda\\hat{I})^{-1}$). Discuss the results of these variances as functions of $\\lambda$. In particular, try to link your discussion with the discussion in Hastie *et al.* and their figures 3.10 and 3.11. **Scikit-Learn** may not provide the variance of the parameters $\\beta$. This needs to be checked. With your own code you can however do so." ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.14" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index a583f36cb..11da16c5c 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -6,10 +6,8 @@ DATE: today !split ===== Plans for week 36 ===== -* Thursday: Summary from last week on discussion of SVD, Ridge and Lasso linear regression. - * "Video of Lecture":"https://youtu.be/qn_BAVhMD8U" -* Friday: Linear Regression and links with Statistics, Resampling methods and presentation of first project - * "Video of Lecture":"https://youtu.be/_CPGg0JYH8M" +* Summary from last week on discussion of SVD, Ridge and Lasso linear regression. +* Linear Regression and links with Statistics, Resampling methods and presentation of first project Recommended Reading: o Lectures on Regression @@ -17,9 +15,6 @@ o Bishop 1.1, 1.2, 2.1, 2.2, 2.3 and 3.1 o Hastie et al chapter 3 -!split -===== Thursday September 8 ===== - !split ===== Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples =====