lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updated txt

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-17 07:29:28 +02:00
parent 2eba427d80
commit 1800607d04
32 changed files with 2426 additions and 267 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter2.html
+371 -63
View File
@@ -157,6 +157,11 @@
6. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapteroptimization.html">
7. Optimization, the central part of any Machine Learning algortithm
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
8. Support Vector Machines, overarching aims
@@ -1101,10 +1106,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.1831277634928002
3.3041320306136366
[[1.03929932 3.07670437]
[3.07670437 9.7701384 ]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.055529303095955385
3.9154537458386387
[[1.0003451 2.94327895]
[2.94327895 9.80021057]]
</pre></div>
</div>
</div>
@@ -1141,10 +1146,10 @@ a more brute force way. Here we scale the mean values for each column of the des
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.09456011349477329
1.6600855222624895
[[1. 0.70183798]
[0.70183798 1. ]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.095786152583691
1.5864689451163851
[[1. 0.6713619]
[0.6713619 1. ]]
</pre></div>
</div>
</div>
@@ -1174,30 +1179,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 0.54316925 0.40968888]
[-1.8393131 -5.8788431 ]
[ 0.24613822 0.9957744 ]
[ 1.38892672 3.93679587]
[ 0.06540809 -0.84008474]
[ 0.49841285 2.31081134]
[-0.49323032 -1.25263916]
[ 0.08380813 -0.7925146 ]
[ 0.64078247 2.92814088]
[-1.13410232 -1.81712976]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-0.10979444 1.25912505]
[ 2.17273709 7.48017006]
[ 0.31520842 0.66152576]
[-0.07076926 -0.44655382]
[ 0.24348403 0.38561052]
[-0.57984245 -0.86011441]
[ 0.42375621 -1.21244261]
[-0.2407922 0.56831157]
[-2.23691315 -6.6936767 ]
[ 0.08292574 -1.14195542]]
0 1
0 0.543169 0.409689
1 -1.839313 -5.878843
2 0.246138 0.995774
3 1.388927 3.936796
4 0.065408 -0.840085
5 0.498413 2.310811
6 -0.493230 -1.252639
7 0.083808 -0.792515
8 0.640782 2.928141
9 -1.134102 -1.817130
0 -0.109794 1.259125
1 2.172737 7.480170
2 0.315208 0.661526
3 -0.070769 -0.446554
4 0.243484 0.385611
5 -0.579842 -0.860114
6 0.423756 -1.212443
7 -0.240792 0.568312
8 -2.236913 -6.693677
9 0.082926 -1.141955
0 1
0 1.000000 0.942726
1 0.942726 1.000000
0 1.000000 0.931066
1 0.931066 1.000000
</pre></div>
</div>
</div>
@@ -1254,37 +1259,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.0 0.080633 0.085811 0.085748 0.084826 0.083221 0.080341 0.078342
2 0.0 0.085811 0.092659 0.093248 0.092932 0.091659 0.088414 0.086561
3 0.0 0.085748 0.093248 0.096639 0.096802 0.095941 0.093928 0.092288
4 0.0 0.084826 0.092932 0.096802 0.097346 0.096763 0.094774 0.093328
5 0.0 0.083221 0.091659 0.095941 0.096763 0.096406 0.094513 0.093236
6 0.0 0.080341 0.088414 0.093928 0.094774 0.094513 0.093573 0.092375
7 0.0 0.078342 0.086561 0.092288 0.093328 0.093236 0.092375 0.091318
8 0.0 0.076213 0.084461 0.090355 0.091535 0.091578 0.090817 0.089881
9 0.0 0.074037 0.082239 0.088258 0.089540 0.089693 0.089039 0.088209
10 0.0 0.073546 0.081474 0.088137 0.089326 0.089437 0.089352 0.088484
11 0.0 0.071453 0.079337 0.086052 0.087334 0.087544 0.087526 0.086757
12 0.0 0.069371 0.077163 0.083905 0.085253 0.085544 0.085598 0.084916
13 0.0 0.067326 0.074996 0.081744 0.083138 0.083495 0.083622 0.083015
14 0.0 0.065334 0.072862 0.079597 0.081024 0.081435 0.081632 0.081092
1 0.0 0.090107 0.080265 0.095180 0.087148 0.079801 0.089928 0.082666
2 0.0 0.080265 0.073052 0.087032 0.080565 0.074557 0.083797 0.077645
3 0.0 0.095180 0.087032 0.106946 0.099079 0.091756 0.104887 0.097150
4 0.0 0.087148 0.080565 0.099079 0.092378 0.086076 0.098028 0.091253
5 0.0 0.079801 0.074557 0.091756 0.086076 0.080678 0.091542 0.085627
6 0.0 0.089928 0.083797 0.104887 0.098028 0.091542 0.105387 0.098187
7 0.0 0.082666 0.077645 0.097150 0.091253 0.085627 0.098187 0.091855
8 0.0 0.076121 0.072046 0.090099 0.085041 0.080173 0.091566 0.086006
9 0.0 0.070212 0.066949 0.083672 0.079351 0.075151 0.085484 0.080612
10 0.0 0.083337 0.078717 0.099609 0.093725 0.088085 0.101781 0.095275
11 0.0 0.076774 0.072995 0.092276 0.087205 0.082303 0.094708 0.088983
12 0.0 0.070866 0.067811 0.085627 0.081270 0.077021 0.088260 0.083228
13 0.0 0.065545 0.063114 0.079598 0.075869 0.072197 0.082382 0.077968
14 0.0 0.060748 0.058856 0.074130 0.070955 0.067793 0.077026 0.073162
8 9 10 11 12 13 14
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.076213 0.074037 0.073546 0.071453 0.069371 0.067326 0.065334
2 0.084461 0.082239 0.081474 0.079337 0.077163 0.074996 0.072862
3 0.090355 0.088258 0.088137 0.086052 0.083905 0.081744 0.079597
4 0.091535 0.089540 0.089326 0.087334 0.085253 0.083138 0.081024
5 0.091578 0.089693 0.089437 0.087544 0.085544 0.083495 0.081435
6 0.090817 0.089039 0.089352 0.087526 0.085598 0.083622 0.081632
7 0.089881 0.088209 0.088484 0.086757 0.084916 0.083015 0.081092
8 0.088556 0.086985 0.087242 0.085610 0.083855 0.082032 0.080182
9 0.086985 0.085510 0.085763 0.084221 0.082549 0.080806 0.079029
10 0.087242 0.085763 0.086430 0.084864 0.083177 0.081425 0.079643
11 0.085610 0.084221 0.084864 0.083384 0.081778 0.080101 0.078390
12 0.083855 0.082549 0.083177 0.081778 0.080249 0.078646 0.077005
13 0.082032 0.080806 0.081425 0.080101 0.078646 0.077115 0.075542
14 0.080182 0.079029 0.079643 0.078390 0.077005 0.075542 0.074036
1 0.076121 0.070212 0.083337 0.076774 0.070866 0.065545 0.060748
2 0.072046 0.066949 0.078717 0.072995 0.067811 0.063114 0.058856
3 0.090099 0.083672 0.099609 0.092276 0.085627 0.079598 0.074130
4 0.085041 0.079351 0.093725 0.087205 0.081270 0.075869 0.070955
5 0.080173 0.075151 0.088085 0.082303 0.077021 0.072197 0.067793
6 0.091566 0.085484 0.101781 0.094708 0.088260 0.082382 0.077026
7 0.086006 0.080612 0.095275 0.088983 0.083228 0.077968 0.073162
8 0.080846 0.076068 0.089250 0.083659 0.078529 0.073827 0.069519
9 0.076068 0.071841 0.083682 0.078720 0.074154 0.069956 0.066098
10 0.089250 0.083682 0.099504 0.092936 0.086920 0.081414 0.076377
11 0.083659 0.078720 0.092936 0.087096 0.081732 0.076810 0.072296
12 0.078529 0.074154 0.086920 0.081732 0.076954 0.072557 0.068515
13 0.073827 0.069956 0.081414 0.076810 0.072557 0.068633 0.065016
14 0.069519 0.066098 0.076377 0.072296 0.068515 0.065016 0.061783
</pre></div>
</div>
</div>
@@ -2864,12 +2869,305 @@ Test MSE OLS
<img alt="_images/chapter2_316_1.png" src="_images/chapter2_316_1.png" />
</div>
</div>
<p>How can we understand this? <strong>More text to be added</strong>.</p>
<p>How can we understand this?</p>
<p>Let us write out the values of the coefficients <span class="math notranslate nohighlight">\(\beta_i\)</span> as functions
of the polynomial degree and noise. We will focus only on the Ridge
results and some few selected values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>If we don’t include any noise and run this code for different values
of the polynomial degree, we notice that the results for <span class="math notranslate nohighlight">\(\beta_i\)</span> do
not show great changes from one order to the next. This is an
indication that for higher polynomial orders, our parameters become
less important.</p>
<p>If we however add noise, what happens is that the polynomial fit is
trying to adjust the fit to traverse in the best possible way all data
points. This can lead to large fluctuations in the parameters
<span class="math notranslate nohighlight">\(\beta_i\)</span> as functions of polynomial order. It will also be reflected
in a larger value of the variance of each parameter <span class="math notranslate nohighlight">\(\beta_i\)</span>. What
Ridge regression (and Lasso as well) are doing then is to try to
quench the fluctuations in the parameters of <span class="math notranslate nohighlight">\(\beta_i\)</span> which have a
large variance (normally for higher orders in the polynomial).</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">display</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">linear_model</span>
<span class="c1"># Make data set.</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">Maxpolydegree</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="n">Maxpolydegree</span><span class="p">))</span>
<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
<span class="k">for</span> <span class="n">polydegree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">Maxpolydegree</span><span class="p">):</span>
<span class="k">for</span> <span class="n">degree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">polydegree</span><span class="p">):</span>
<span class="n">X</span><span class="p">[:,</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="p">(</span><span class="n">degree</span><span class="p">)</span>
<span class="c1"># We split the data in test and training data</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="c1"># Decide which values of lambda to use</span>
<span class="n">nlambdas</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">lambdas</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">logspace</span><span class="p">(</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">nlambdas</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">):</span>
<span class="n">lmb</span> <span class="o">=</span> <span class="n">lambdas</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="c1"># Make the fit using Ridge only</span>
<span class="n">RegRidge</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">Ridge</span><span class="p">(</span><span class="n">lmb</span><span class="p">,</span><span class="n">fit_intercept</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">RegRidge</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span><span class="n">y_train</span><span class="p">)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ypredictRidge</span> <span class="o">=</span> <span class="n">RegRidge</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">Coeffs</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">RegRidge</span><span class="o">.</span><span class="n">coef_</span><span class="p">)</span>
<span class="n">BetaValues</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">Coeffs</span><span class="p">)</span>
<span class="n">BetaValues</span><span class="o">.</span><span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;beta&#39;</span><span class="p">]</span>
<span class="n">display</span><span class="p">(</span><span class="n">BetaValues</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.986699</td>
</tr>
<tr>
<th>1</th>
<td>-0.606760</td>
</tr>
<tr>
<th>2</th>
<td>1.280573</td>
</tr>
<tr>
<th>3</th>
<td>-0.850164</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div><div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.978553</td>
</tr>
<tr>
<th>1</th>
<td>-0.511888</td>
</tr>
<tr>
<th>2</th>
<td>1.051418</td>
</tr>
<tr>
<th>3</th>
<td>-0.701370</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div><div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.946957</td>
</tr>
<tr>
<th>1</th>
<td>-0.162246</td>
</tr>
<tr>
<th>2</th>
<td>0.221921</td>
</tr>
<tr>
<th>3</th>
<td>-0.167787</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div><div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.906747</td>
</tr>
<tr>
<th>1</th>
<td>0.017665</td>
</tr>
<tr>
<th>2</th>
<td>-0.029483</td>
</tr>
<tr>
<th>3</th>
<td>-0.053849</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div><div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.718165</td>
</tr>
<tr>
<th>1</th>
<td>0.156956</td>
</tr>
<tr>
<th>2</th>
<td>0.040102</td>
</tr>
<tr>
<th>3</th>
<td>-0.001880</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div></div>
</div>
<p>As an exercise, repeat these calculations with ordinary least squares
only with and without noise. Calculate thereafter the variance of the
parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> as function of polynomial order and of the added
noise. Here we recommend to use <span class="math notranslate nohighlight">\(\sigma^2=1\)</span> as variance for the
added noise (which follows a normal distribution with mean value zero).
Comment your results. If you have a large noise term, do the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> vary more as function
model complexity? And what about their variance?</p>
</div>
<div class="section" id="linking-bayes-theorem-with-ridge-and-lasso-regression">
<h2><span class="section-number">4.14. </span>Linking Bayes’ Theorem with Ridge and Lasso Regression<a class="headerlink" href="#linking-bayes-theorem-with-ridge-and-lasso-regression" title="Permalink to this headline"></a></h2>
<p>Using Bayes’ theorem we can gain a better intuition about Ridge and Lasso regression.</p>
<p>For ordinary least squares we postulated that the maximum likelihood for the doamin of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> (one-dimensional case)</p>
<p>We have seen that Ridge regression suppresses those features which
have a small singular value. This corresponds to a feature which exhibits
a large variance in the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span>.
Our analysis hitherto has been based on linear algebra. To add to our intuition, we will use
Bayes’ theorem in order to deepen our understanding of Ridge and Lasso regression.</p>
<p>For ordinary least squares we postulated that the maximum likelihood for the domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> (one-dimensional case)</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})],
@@ -2893,7 +3191,10 @@ p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol
<p>With the posterior probability defined by a likelihood which we have
already modeled and an unknown prior, we are now ready to make
additional models for the prior.</p>
<p>We can, based on our discussions of the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and the mean value, assume that the prior for the values <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is given by a Gaussian with mean value zero and variance <span class="math notranslate nohighlight">\(\tau^2\)</span>, that is</p>
<p>We can, based on our discussions of the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and
the mean value, assume that the prior for the values <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is
given by a Gaussian with mean value zero and variance <span class="math notranslate nohighlight">\(\tau^2\)</span>, that
is</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}.
@@ -2906,7 +3207,7 @@ p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\si
<p>We can now optimize this quantity with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. As we
did for OLS, this is most conveniently done by taking the negative
logarithm of the posterior probability. Doing so and leaving out the
constants terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
@@ -2940,6 +3241,13 @@ C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{
C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
\]</div>
<p>which is our Lasso cost function!</p>
<p>Plotting these prior functions shows us that we can use the parameter
<span class="math notranslate nohighlight">\(\lambda\)</span> to shrink or increase the role of a given parameter
<span class="math notranslate nohighlight">\(\beta_j\)</span>. The variance for the Laplace distribution is
<span class="math notranslate nohighlight">\(2\tau^2=1/\lambda\)</span> while for the Gaussian distribution it is
<span class="math notranslate nohighlight">\(\sigma^2=1/(2\lambda)\)</span>. Thus, increasing the variance means
decreasing <span class="math notranslate nohighlight">\(\lambda\)</span> and shrinking the variance means increasing
<span class="math notranslate nohighlight">\(\lamdbda\)</span>. When we increase <span class="math notranslate nohighlight">\(\lambda\)</span>, this corresponds to shrinking the role of less important features (small singular values).</p>
</div>
</div>