update of jupyter book
This commit is contained in:
Morten Hjorth-Jensen
@@ -478,7 +478,8 @@ const thebe_selector_output = ".output, .cell_output"
|
||||
|
||||
<div>
|
||||
|
||||
<div class="tex2jax_ignore mathjax_ignore section" id="ridge-and-lasso-regression">
|
||||
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
|
||||
doconce format html chapter2.do.txt --><div class="tex2jax_ignore mathjax_ignore section" id="ridge-and-lasso-regression">
|
||||
<h1><span class="section-number">4. </span>Ridge and Lasso Regression<a class="headerlink" href="#ridge-and-lasso-regression" title="Permalink to this headline">¶</a></h1>
|
||||
<div class="section" id="mathematical-interpretation-of-ordinary-least-squares">
|
||||
<h2><span class="section-number">4.1. </span>Mathematical Interpretation of Ordinary Least Squares<a class="headerlink" href="#mathematical-interpretation-of-ordinary-least-squares" title="Permalink to this headline">¶</a></h2>
|
||||
@@ -850,7 +851,7 @@ It is used for the calculation of the inverse for singular or near singular matr
|
||||
\[
|
||||
\boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T,
|
||||
\]</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{D}_{\mathrm{PI}}\)</span> can be calculated by creating a diagonal matrix from <span class="math notranslate nohighlight">\(\boldsymbol{Sigma}\)</span> where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.</p>
|
||||
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{D}_{\mathrm{PI}}\)</span> can be calculated by creating a diagonal matrix from <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
@@ -983,11 +984,6 @@ decomposition of the design matrix.</p>
|
||||
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
|
||||
\]</div>
|
||||
<p>We define <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2\)</span> which is a diagonal matrix containing only the singular values squared. It has dimensionality <span class="math notranslate nohighlight">\(p \times p\)</span>.</p>
|
||||
<p>This means, using the orthogonality of <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>, that we get</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{X}^T\boldsymbol{X}=\tilde{\boldsymbol{\Sigma}}^2.
|
||||
\]</div>
|
||||
<p>We can now insert the result for the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> into our equation for ordinary least squares where</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
@@ -996,9 +992,9 @@ decomposition of the design matrix.</p>
|
||||
<p>and using our SVD decomposition of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\tilde{\boldsymbol{\Sigma}}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y},
|
||||
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\left(\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^{2}(\boldsymbol{V}^T\right)^{-1}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y},
|
||||
\]</div>
|
||||
<p>which gives us, using the orthogonality of the matrices <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>,</p>
|
||||
<p>which gives us, using the orthogonality of the matrices <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>,,</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y},
|
||||
@@ -1214,10 +1210,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.13876586436927824
|
||||
3.722047011333792
|
||||
[[ 1.233528 3.58428804]
|
||||
[ 3.58428804 11.47942814]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.01591355407242949
|
||||
3.6808538439837775
|
||||
[[ 0.96390357 2.99157584]
|
||||
[ 2.99157584 10.31120247]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1254,10 +1250,10 @@ a more brute force way. Here we scale the mean values for each column of the des
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08464758160254343
|
||||
1.8503720991789538
|
||||
[[1. 0.65626043]
|
||||
[0.65626043 1. ]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08612280083325631
|
||||
1.6149274949460215
|
||||
[[1. 0.66934291]
|
||||
[0.66934291 1. ]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1287,30 +1283,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-1.41876267 -4.93248252]
|
||||
[ 1.83687444 5.28097861]
|
||||
[ 0.37429133 0.59766 ]
|
||||
[ 0.59159438 1.71869727]
|
||||
[-0.80315282 -0.89348922]
|
||||
[-0.38748219 -2.12288563]
|
||||
[-2.08917679 -5.64933923]
|
||||
[ 0.27803645 0.89944994]
|
||||
[ 1.23703839 3.2321528 ]
|
||||
[ 0.38073947 1.86925797]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-1.04649105 -2.92658312]
|
||||
[ 0.45985488 1.43876695]
|
||||
[-0.41081513 -1.96426825]
|
||||
[ 1.75703965 4.88736621]
|
||||
[ 1.02698605 3.59304008]
|
||||
[-0.71348713 -1.98249059]
|
||||
[-0.22685646 0.37866422]
|
||||
[-0.90559087 -1.31597731]
|
||||
[-0.60349429 -3.47245463]
|
||||
[ 0.66285434 1.36393643]]
|
||||
0 1
|
||||
0 -1.418763 -4.932483
|
||||
1 1.836874 5.280979
|
||||
2 0.374291 0.597660
|
||||
3 0.591594 1.718697
|
||||
4 -0.803153 -0.893489
|
||||
5 -0.387482 -2.122886
|
||||
6 -2.089177 -5.649339
|
||||
7 0.278036 0.899450
|
||||
8 1.237038 3.232153
|
||||
9 0.380739 1.869258
|
||||
0 -1.046491 -2.926583
|
||||
1 0.459855 1.438767
|
||||
2 -0.410815 -1.964268
|
||||
3 1.757040 4.887366
|
||||
4 1.026986 3.593040
|
||||
5 -0.713487 -1.982491
|
||||
6 -0.226856 0.378664
|
||||
7 -0.905591 -1.315977
|
||||
8 -0.603494 -3.472455
|
||||
9 0.662854 1.363936
|
||||
0 1
|
||||
0 1.000000 0.977418
|
||||
1 0.977418 1.000000
|
||||
0 1.000000 0.948641
|
||||
1 0.948641 1.000000
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1367,37 +1363,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
|
||||
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
|
||||
1 0.0 0.081253 0.083015 0.080297 0.079157 0.078040 0.071112 0.069847
|
||||
2 0.0 0.083015 0.086807 0.084909 0.084904 0.084702 0.076955 0.076338
|
||||
3 0.0 0.080297 0.084909 0.084414 0.084862 0.085018 0.077793 0.077402
|
||||
4 0.0 0.079157 0.084904 0.084862 0.086076 0.086872 0.079281 0.079383
|
||||
5 0.0 0.078040 0.084702 0.085018 0.086872 0.088212 0.080319 0.080847
|
||||
6 0.0 0.071112 0.076955 0.077793 0.079281 0.080319 0.073719 0.074034
|
||||
7 0.0 0.069847 0.076338 0.077402 0.079383 0.080847 0.074034 0.074693
|
||||
8 0.0 0.068735 0.075760 0.077003 0.079405 0.081233 0.074237 0.075195
|
||||
9 0.0 0.067769 0.075241 0.076628 0.079389 0.081533 0.074375 0.075594
|
||||
10 0.0 0.062136 0.068318 0.069888 0.071921 0.073449 0.067622 0.068376
|
||||
11 0.0 0.061149 0.067723 0.069408 0.071767 0.073583 0.067612 0.068608
|
||||
12 0.0 0.060309 0.067213 0.068991 0.071632 0.073699 0.067599 0.068806
|
||||
13 0.0 0.059601 0.066789 0.068642 0.071529 0.073816 0.067597 0.068992
|
||||
14 0.0 0.059013 0.066449 0.068363 0.071467 0.073949 0.067620 0.069181
|
||||
1 0.0 0.084846 0.071547 0.086679 0.078725 0.071480 0.079609 0.073320
|
||||
2 0.0 0.071547 0.061716 0.073647 0.067908 0.062640 0.068417 0.063857
|
||||
3 0.0 0.086679 0.073647 0.094619 0.086262 0.078697 0.090356 0.083579
|
||||
4 0.0 0.078725 0.067908 0.086262 0.079483 0.073303 0.082874 0.077375
|
||||
5 0.0 0.071480 0.062640 0.078697 0.073303 0.068345 0.076121 0.071741
|
||||
6 0.0 0.079609 0.068417 0.090356 0.082874 0.076121 0.088460 0.082267
|
||||
7 0.0 0.073320 0.063857 0.083579 0.077375 0.071741 0.082267 0.077131
|
||||
8 0.0 0.067787 0.059824 0.077620 0.072521 0.067856 0.076821 0.072597
|
||||
9 0.0 0.062888 0.056235 0.072352 0.068210 0.064388 0.072008 0.068573
|
||||
10 0.0 0.071906 0.062551 0.083694 0.077291 0.071515 0.083361 0.077978
|
||||
11 0.0 0.066654 0.058715 0.077948 0.072611 0.067767 0.078042 0.073548
|
||||
12 0.0 0.062061 0.055344 0.072918 0.068500 0.064461 0.073380 0.069654
|
||||
13 0.0 0.058033 0.052372 0.068505 0.064880 0.061538 0.069285 0.066223
|
||||
14 0.0 0.054491 0.049747 0.064623 0.061685 0.058948 0.065680 0.063192
|
||||
|
||||
8 9 10 11 12 13 14
|
||||
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
|
||||
1 0.068735 0.067769 0.062136 0.061149 0.060309 0.059601 0.059013
|
||||
2 0.075760 0.075241 0.068318 0.067723 0.067213 0.066789 0.066449
|
||||
3 0.077003 0.076628 0.069888 0.069408 0.068991 0.068642 0.068363
|
||||
4 0.079405 0.079389 0.071921 0.071767 0.071632 0.071529 0.071467
|
||||
5 0.081233 0.081533 0.073449 0.073583 0.073699 0.073816 0.073949
|
||||
6 0.074237 0.074375 0.067622 0.067612 0.067599 0.067597 0.067620
|
||||
7 0.075195 0.075594 0.068376 0.068608 0.068806 0.068992 0.069181
|
||||
8 0.075959 0.076589 0.068965 0.069409 0.069796 0.070150 0.070491
|
||||
9 0.076589 0.077425 0.069441 0.070074 0.070630 0.071136 0.071614
|
||||
10 0.068965 0.069441 0.063052 0.063364 0.063631 0.063874 0.064110
|
||||
11 0.069409 0.070074 0.063364 0.063851 0.064274 0.064658 0.065020
|
||||
12 0.069796 0.070630 0.063631 0.064274 0.064838 0.065348 0.065826
|
||||
13 0.070150 0.071136 0.063874 0.064658 0.065348 0.065974 0.066558
|
||||
14 0.070491 0.071614 0.064110 0.065020 0.065826 0.066558 0.067240
|
||||
1 0.067787 0.062888 0.071906 0.066654 0.062061 0.058033 0.054491
|
||||
2 0.059824 0.056235 0.062551 0.058715 0.055344 0.052372 0.049747
|
||||
3 0.077620 0.072352 0.083694 0.077948 0.072918 0.068505 0.064623
|
||||
4 0.072521 0.068210 0.077291 0.072611 0.068500 0.064880 0.061685
|
||||
5 0.067856 0.064388 0.071515 0.067767 0.064461 0.061538 0.058948
|
||||
6 0.076821 0.072008 0.083361 0.078042 0.073380 0.069285 0.065680
|
||||
7 0.072597 0.068573 0.077978 0.073548 0.069654 0.066223 0.063192
|
||||
8 0.068852 0.065514 0.073238 0.069578 0.066348 0.063493 0.060963
|
||||
9 0.065514 0.062773 0.069044 0.066051 0.063401 0.061048 0.058955
|
||||
10 0.073238 0.069044 0.079558 0.074881 0.070775 0.067162 0.063977
|
||||
11 0.069578 0.066051 0.074881 0.070959 0.067506 0.064459 0.061765
|
||||
12 0.066348 0.063401 0.070775 0.067506 0.064618 0.062062 0.059795
|
||||
13 0.063493 0.061048 0.067162 0.064459 0.062062 0.059933 0.058038
|
||||
14 0.060963 0.058955 0.063977 0.061765 0.059795 0.058038 0.056468
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1439,7 +1435,9 @@ x_{01}x_{00}+x_{11}x_{10} & x_{01}^2+x_{11}^2\\
|
||||
\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\
|
||||
\end{bmatrix},
|
||||
\end{split}\]</div>
|
||||
<p>where we wrote $<span class="math notranslate nohighlight">\(\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]\)</span><span class="math notranslate nohighlight">\( to indicate that this is the covariance of the vectors \)</span>\boldsymbol{x}<span class="math notranslate nohighlight">\( of the design/feature matrix \)</span>\boldsymbol{X}$.</p>
|
||||
<p>where we wrote <span class="math notranslate nohighlight">\(\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1]=\boldsymbol{C}[\boldsymbol{x}]\)</span> to indicate
|
||||
that this is the covariance of the vectors <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> of the
|
||||
design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
|
||||
<p>It is easy to generalize this to a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>.</p>
|
||||
</div>
|
||||
<div class="section" id="linking-with-the-svd">
|
||||
@@ -1880,7 +1878,7 @@ Training MSE for OLS
|
||||
3.0
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter2_245_1.png" src="_images/chapter2_245_1.png" />
|
||||
<img alt="_images/chapter2_254_1.png" src="_images/chapter2_254_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
|
||||
@@ -2156,7 +2154,7 @@ Training MSE for OLS
|
||||
[ 0. -0.]
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter2_247_1.png" src="_images/chapter2_247_1.png" />
|
||||
<img alt="_images/chapter2_256_1.png" src="_images/chapter2_256_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>We bring then back our exponential function example and study all
|
||||
@@ -2259,305 +2257,14 @@ Test MSE OLS
|
||||
0.008675369724975977
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter2_249_1.png" src="_images/chapter2_249_1.png" />
|
||||
<img alt="_images/chapter2_258_1.png" src="_images/chapter2_258_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>Both these example send a clear message. The addition of a
|
||||
shrinkage/regularization term implies that we need to perform a search
|
||||
for the optimal values of <span class="math notranslate nohighlight">\(\lambda\)</span>. We will see this throughout these
|
||||
series of lectures.</p>
|
||||
<p>As a small addendum, we note that you can also solve this problem using the convex optimization package <a class="reference external" href="https://cvxopt.org/examples/mlbook/l1regls.html">CVXOPT</a>. This requires, in addition to having installed <strong>CVXOPT</strong>, you need to download the file <em><a class="reference external" href="http://l1regl.py">l1regl.py</a></em>.
|
||||
The following code example solves the simpler problem we discussed above, where we have added the latter python file.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cvxopt</span> <span class="kn">import</span> <span class="n">matrix</span><span class="p">,</span> <span class="n">spdiag</span><span class="p">,</span> <span class="n">mul</span><span class="p">,</span> <span class="n">div</span><span class="p">,</span> <span class="n">sqrt</span><span class="p">,</span> <span class="n">normal</span><span class="p">,</span> <span class="n">setseed</span>
|
||||
<span class="kn">from</span> <span class="nn">cvxopt</span> <span class="kn">import</span> <span class="n">blas</span><span class="p">,</span> <span class="n">lapack</span><span class="p">,</span> <span class="n">solvers</span><span class="p">,</span> <span class="n">sparse</span><span class="p">,</span> <span class="n">spmatrix</span>
|
||||
<span class="kn">import</span> <span class="nn">math</span>
|
||||
|
||||
<span class="k">try</span><span class="p">:</span>
|
||||
<span class="kn">import</span> <span class="nn">mosek</span>
|
||||
<span class="kn">import</span> <span class="nn">sys</span>
|
||||
<span class="n">__MOSEK</span> <span class="o">=</span> <span class="kc">True</span>
|
||||
<span class="k">except</span><span class="p">:</span> <span class="n">__MOSEK</span> <span class="o">=</span> <span class="kc">False</span>
|
||||
|
||||
<span class="k">if</span> <span class="n">__MOSEK</span><span class="p">:</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">l1regls_mosek</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
|
||||
<span class="sd">"""</span>
|
||||
|
||||
<span class="sd"> Returns the solution of l1-norm regularized least-squares problem</span>
|
||||
|
||||
<span class="sd"> minimize || A*x - b ||_2^2 + e'*u</span>
|
||||
|
||||
<span class="sd"> subject to -u <= x <= u</span>
|
||||
|
||||
<span class="sd"> """</span>
|
||||
|
||||
<span class="n">m</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="n">A</span><span class="o">.</span><span class="n">size</span>
|
||||
|
||||
<span class="n">env</span> <span class="o">=</span> <span class="n">mosek</span><span class="o">.</span><span class="n">Env</span><span class="p">()</span>
|
||||
<span class="n">task</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">Task</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">)</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">set_Stream</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">streamtype</span><span class="o">.</span><span class="n">log</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
|
||||
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">appendvars</span><span class="p">(</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">)</span> <span class="c1"># number of variables</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">appendcons</span><span class="p">(</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">)</span> <span class="c1"># number of constraints</span>
|
||||
|
||||
<span class="c1"># input quadratic objective</span>
|
||||
<span class="n">Q</span> <span class="o">=</span> <span class="n">matrix</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="n">n</span><span class="p">))</span>
|
||||
<span class="n">blas</span><span class="o">.</span><span class="n">syrk</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">Q</span><span class="p">,</span> <span class="n">alpha</span> <span class="o">=</span> <span class="mf">2.0</span><span class="p">,</span> <span class="n">trans</span><span class="o">=</span><span class="s1">'T'</span><span class="p">)</span>
|
||||
|
||||
<span class="n">I</span> <span class="o">=</span> <span class="p">[]</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
|
||||
<span class="n">I</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="n">n</span><span class="p">))</span>
|
||||
|
||||
<span class="n">J</span> <span class="o">=</span> <span class="p">[]</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
|
||||
<span class="n">J</span><span class="o">.</span><span class="n">extend</span><span class="p">((</span><span class="n">n</span><span class="o">-</span><span class="n">i</span><span class="p">)</span><span class="o">*</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
|
||||
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putqobj</span><span class="p">(</span><span class="n">I</span><span class="p">,</span> <span class="n">J</span><span class="p">,</span> <span class="nb">list</span><span class="p">(</span><span class="n">Q</span><span class="p">[</span><span class="n">matrix</span><span class="p">(</span><span class="n">I</span><span class="p">)</span> <span class="o">+</span> <span class="n">matrix</span><span class="p">(</span><span class="n">J</span><span class="p">)</span><span class="o">*</span><span class="n">n</span><span class="p">]))</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putclist</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">),</span> <span class="nb">list</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="o">*</span><span class="n">A</span><span class="o">.</span><span class="n">T</span><span class="o">*</span><span class="n">b</span><span class="p">)</span> <span class="o">+</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">1.0</span><span class="p">])</span> <span class="c1"># setup linear objective</span>
|
||||
|
||||
<span class="c1"># input constraint matrix row by row</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putarow</span><span class="p">(</span> <span class="n">i</span><span class="p">,</span> <span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">n</span><span class="o">+</span><span class="n">i</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.0</span><span class="p">])</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putarow</span><span class="p">(</span> <span class="n">n</span><span class="o">+</span><span class="n">i</span><span class="p">,</span> <span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">n</span><span class="o">+</span><span class="n">i</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">])</span>
|
||||
|
||||
<span class="c1"># setup bounds on constraints</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putboundslice</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">accmode</span><span class="o">.</span><span class="n">con</span><span class="p">,</span>
|
||||
<span class="mi">0</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="n">mosek</span><span class="o">.</span><span class="n">boundkey</span><span class="o">.</span><span class="n">up</span><span class="p">],</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">],</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">])</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putboundslice</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">accmode</span><span class="o">.</span><span class="n">con</span><span class="p">,</span>
|
||||
<span class="n">n</span><span class="p">,</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">,</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="n">mosek</span><span class="o">.</span><span class="n">boundkey</span><span class="o">.</span><span class="n">lo</span><span class="p">],</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">],</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">])</span>
|
||||
|
||||
<span class="c1"># setup variable bounds</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putboundslice</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">accmode</span><span class="o">.</span><span class="n">var</span><span class="p">,</span>
|
||||
<span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">,</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="n">mosek</span><span class="o">.</span><span class="n">boundkey</span><span class="o">.</span><span class="n">fr</span><span class="p">],</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">],</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">])</span>
|
||||
|
||||
<span class="c1"># optimize the task</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putobjsense</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">objsense</span><span class="o">.</span><span class="n">minimize</span><span class="p">)</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">optimize</span><span class="p">()</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">solutionsummary</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">streamtype</span><span class="o">.</span><span class="n">log</span><span class="p">)</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">]</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">getsolutionslice</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">soltype</span><span class="o">.</span><span class="n">itr</span><span class="p">,</span> <span class="n">mosek</span><span class="o">.</span><span class="n">solitem</span><span class="o">.</span><span class="n">xx</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">matrix</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">l1regls_mosek2</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
|
||||
<span class="sd">"""</span>
|
||||
|
||||
<span class="sd"> Returns the solution of l1-norm regularized least-squares problem</span>
|
||||
|
||||
<span class="sd"> minimize w'*w + e'*u</span>
|
||||
|
||||
<span class="sd"> subject to -u <= x <= u</span>
|
||||
|
||||
<span class="sd"> A*x - w = b</span>
|
||||
|
||||
<span class="sd"> """</span>
|
||||
|
||||
<span class="n">m</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="n">A</span><span class="o">.</span><span class="n">size</span>
|
||||
|
||||
<span class="n">env</span> <span class="o">=</span> <span class="n">mosek</span><span class="o">.</span><span class="n">Env</span><span class="p">()</span>
|
||||
<span class="n">task</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">Task</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">)</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">set_Stream</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">streamtype</span><span class="o">.</span><span class="n">log</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
|
||||
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">appendvars</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span> <span class="o">+</span> <span class="n">m</span><span class="p">)</span> <span class="c1"># number of variables</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">appendcons</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span> <span class="o">+</span> <span class="n">m</span><span class="p">)</span> <span class="c1"># number of constraints</span>
|
||||
|
||||
<span class="c1"># input quadratic objective</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putqobj</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">,</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">m</span><span class="p">),</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">,</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">m</span><span class="p">),</span> <span class="n">m</span><span class="o">*</span><span class="p">[</span><span class="mf">2.0</span><span class="p">])</span>
|
||||
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putclist</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">m</span><span class="p">),</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">]</span> <span class="o">+</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">1.0</span><span class="p">]</span> <span class="o">+</span> <span class="n">m</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">])</span> <span class="c1"># setup linear objective</span>
|
||||
|
||||
<span class="c1"># input constraint matrix row by row</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putarow</span><span class="p">(</span> <span class="n">i</span><span class="p">,</span> <span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">n</span><span class="o">+</span><span class="n">i</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.0</span><span class="p">])</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putarow</span><span class="p">(</span> <span class="n">n</span><span class="o">+</span><span class="n">i</span><span class="p">,</span> <span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">n</span><span class="o">+</span><span class="n">i</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">])</span>
|
||||
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putarow</span><span class="p">(</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">i</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">i</span><span class="p">],</span> <span class="nb">list</span><span class="p">(</span><span class="n">A</span><span class="p">[</span><span class="n">i</span><span class="p">,:])</span> <span class="o">+</span> <span class="p">[</span><span class="o">-</span><span class="mf">1.0</span><span class="p">])</span>
|
||||
|
||||
<span class="c1"># setup bounds on constraints</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putboundslice</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">accmode</span><span class="o">.</span><span class="n">con</span><span class="p">,</span>
|
||||
<span class="mi">0</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="n">mosek</span><span class="o">.</span><span class="n">boundkey</span><span class="o">.</span><span class="n">up</span><span class="p">],</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">],</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">])</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putboundslice</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">accmode</span><span class="o">.</span><span class="n">con</span><span class="p">,</span>
|
||||
<span class="n">n</span><span class="p">,</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">,</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="n">mosek</span><span class="o">.</span><span class="n">boundkey</span><span class="o">.</span><span class="n">lo</span><span class="p">],</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">],</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">])</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putboundslice</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">accmode</span><span class="o">.</span><span class="n">con</span><span class="p">,</span>
|
||||
<span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">,</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">m</span><span class="p">,</span> <span class="n">m</span><span class="o">*</span><span class="p">[</span><span class="n">mosek</span><span class="o">.</span><span class="n">boundkey</span><span class="o">.</span><span class="n">fx</span><span class="p">],</span> <span class="nb">list</span><span class="p">(</span><span class="n">b</span><span class="p">),</span> <span class="nb">list</span><span class="p">(</span><span class="n">b</span><span class="p">))</span>
|
||||
|
||||
<span class="c1"># setup variable bounds</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putboundslice</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">accmode</span><span class="o">.</span><span class="n">var</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">m</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">m</span><span class="p">)</span><span class="o">*</span><span class="p">[</span><span class="n">mosek</span><span class="o">.</span><span class="n">boundkey</span><span class="o">.</span><span class="n">fr</span><span class="p">],</span>
|
||||
<span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">m</span><span class="p">)</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">],</span> <span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="n">m</span><span class="p">)</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">])</span>
|
||||
|
||||
<span class="c1"># optimize the task</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">putobjsense</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">objsense</span><span class="o">.</span><span class="n">minimize</span><span class="p">)</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">optimize</span><span class="p">()</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">solutionsummary</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">streamtype</span><span class="o">.</span><span class="n">log</span><span class="p">)</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">n</span><span class="o">*</span><span class="p">[</span><span class="mf">0.0</span><span class="p">]</span>
|
||||
<span class="n">task</span><span class="o">.</span><span class="n">getsolutionslice</span><span class="p">(</span><span class="n">mosek</span><span class="o">.</span><span class="n">soltype</span><span class="o">.</span><span class="n">itr</span><span class="p">,</span> <span class="n">mosek</span><span class="o">.</span><span class="n">solitem</span><span class="o">.</span><span class="n">xx</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">matrix</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">l1regls</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
|
||||
<span class="sd">"""</span>
|
||||
<span class="sd"> </span>
|
||||
<span class="sd"> Returns the solution of l1-norm regularized least-squares problem</span>
|
||||
<span class="sd"> </span>
|
||||
<span class="sd"> minimize || A*x - b ||_2^2 + || x ||_1.</span>
|
||||
|
||||
<span class="sd"> """</span>
|
||||
|
||||
<span class="n">m</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="n">A</span><span class="o">.</span><span class="n">size</span>
|
||||
<span class="n">q</span> <span class="o">=</span> <span class="n">matrix</span><span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
|
||||
<span class="n">q</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="o">-</span><span class="mf">2.0</span> <span class="o">*</span> <span class="n">A</span><span class="o">.</span><span class="n">T</span> <span class="o">*</span> <span class="n">b</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">P</span><span class="p">(</span><span class="n">u</span><span class="p">,</span> <span class="n">v</span><span class="p">,</span> <span class="n">alpha</span> <span class="o">=</span> <span class="mf">1.0</span><span class="p">,</span> <span class="n">beta</span> <span class="o">=</span> <span class="mf">0.0</span> <span class="p">):</span>
|
||||
<span class="sd">"""</span>
|
||||
<span class="sd"> v := alpha * 2.0 * [ A'*A, 0; 0, 0 ] * u + beta * v </span>
|
||||
<span class="sd"> """</span>
|
||||
<span class="n">v</span> <span class="o">*=</span> <span class="n">beta</span>
|
||||
<span class="n">v</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">+=</span> <span class="n">alpha</span> <span class="o">*</span> <span class="mf">2.0</span> <span class="o">*</span> <span class="n">A</span><span class="o">.</span><span class="n">T</span> <span class="o">*</span> <span class="p">(</span><span class="n">A</span> <span class="o">*</span> <span class="n">u</span><span class="p">[:</span><span class="n">n</span><span class="p">])</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">G</span><span class="p">(</span><span class="n">u</span><span class="p">,</span> <span class="n">v</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">beta</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">trans</span><span class="o">=</span><span class="s1">'N'</span><span class="p">):</span>
|
||||
<span class="sd">"""</span>
|
||||
<span class="sd"> v := alpha*[I, -I; -I, -I] * u + beta * v (trans = 'N' or 'T')</span>
|
||||
<span class="sd"> """</span>
|
||||
|
||||
<span class="n">v</span> <span class="o">*=</span> <span class="n">beta</span>
|
||||
<span class="n">v</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">+=</span> <span class="n">alpha</span><span class="o">*</span><span class="p">(</span><span class="n">u</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">-</span> <span class="n">u</span><span class="p">[</span><span class="n">n</span><span class="p">:])</span>
|
||||
<span class="n">v</span><span class="p">[</span><span class="n">n</span><span class="p">:]</span> <span class="o">+=</span> <span class="n">alpha</span><span class="o">*</span><span class="p">(</span><span class="o">-</span><span class="n">u</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">-</span> <span class="n">u</span><span class="p">[</span><span class="n">n</span><span class="p">:])</span>
|
||||
|
||||
<span class="n">h</span> <span class="o">=</span> <span class="n">matrix</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
|
||||
|
||||
|
||||
<span class="c1"># Customized solver for the KKT system </span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># [ 2.0*A'*A 0 I -I ] [x[:n] ] [bx[:n] ]</span>
|
||||
<span class="c1"># [ 0 0 -I -I ] [x[n:] ] = [bx[n:] ].</span>
|
||||
<span class="c1"># [ I -I -D1^-1 0 ] [zl[:n]] [bzl[:n]]</span>
|
||||
<span class="c1"># [ -I -I 0 -D2^-1 ] [zl[n:]] [bzl[n:]]</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># where D1 = W['di'][:n]**2, D2 = W['di'][:n]**2.</span>
|
||||
<span class="c1"># </span>
|
||||
<span class="c1"># We first eliminate zl and x[n:]:</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># ( 2*A'*A + 4*D1*D2*(D1+D2)^-1 ) * x[:n] = </span>
|
||||
<span class="c1"># bx[:n] - (D2-D1)*(D1+D2)^-1 * bx[n:] + </span>
|
||||
<span class="c1"># D1 * ( I + (D2-D1)*(D1+D2)^-1 ) * bzl[:n] - </span>
|
||||
<span class="c1"># D2 * ( I - (D2-D1)*(D1+D2)^-1 ) * bzl[n:] </span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># x[n:] = (D1+D2)^-1 * ( bx[n:] - D1*bzl[:n] - D2*bzl[n:] ) </span>
|
||||
<span class="c1"># - (D2-D1)*(D1+D2)^-1 * x[:n] </span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># zl[:n] = D1 * ( x[:n] - x[n:] - bzl[:n] )</span>
|
||||
<span class="c1"># zl[n:] = D2 * (-x[:n] - x[n:] - bzl[n:] ).</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># The first equation has the form</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># (A'*A + D)*x[:n] = rhs</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># and is equivalent to</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># [ D A' ] [ x:n] ] = [ rhs ]</span>
|
||||
<span class="c1"># [ A -I ] [ v ] [ 0 ].</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># It can be solved as </span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># ( A*D^-1*A' + I ) * v = A * D^-1 * rhs</span>
|
||||
<span class="c1"># x[:n] = D^-1 * ( rhs - A'*v ).</span>
|
||||
|
||||
<span class="n">S</span> <span class="o">=</span> <span class="n">matrix</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="p">(</span><span class="n">m</span><span class="p">,</span><span class="n">m</span><span class="p">))</span>
|
||||
<span class="n">Asc</span> <span class="o">=</span> <span class="n">matrix</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="p">(</span><span class="n">m</span><span class="p">,</span><span class="n">n</span><span class="p">))</span>
|
||||
<span class="n">v</span> <span class="o">=</span> <span class="n">matrix</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="p">(</span><span class="n">m</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">Fkkt</span><span class="p">(</span><span class="n">W</span><span class="p">):</span>
|
||||
|
||||
<span class="c1"># Factor </span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># S = A*D^-1*A' + I </span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># where D = 2*D1*D2*(D1+D2)^-1, D1 = d[:n]**-2, D2 = d[n:]**-2.</span>
|
||||
|
||||
<span class="n">d1</span><span class="p">,</span> <span class="n">d2</span> <span class="o">=</span> <span class="n">W</span><span class="p">[</span><span class="s1">'di'</span><span class="p">][:</span><span class="n">n</span><span class="p">]</span><span class="o">**</span><span class="mi">2</span><span class="p">,</span> <span class="n">W</span><span class="p">[</span><span class="s1">'di'</span><span class="p">][</span><span class="n">n</span><span class="p">:]</span><span class="o">**</span><span class="mi">2</span>
|
||||
|
||||
<span class="c1"># ds is square root of diagonal of D</span>
|
||||
<span class="n">ds</span> <span class="o">=</span> <span class="n">math</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="mf">2.0</span><span class="p">)</span> <span class="o">*</span> <span class="n">div</span><span class="p">(</span> <span class="n">mul</span><span class="p">(</span> <span class="n">W</span><span class="p">[</span><span class="s1">'di'</span><span class="p">][:</span><span class="n">n</span><span class="p">],</span> <span class="n">W</span><span class="p">[</span><span class="s1">'di'</span><span class="p">][</span><span class="n">n</span><span class="p">:]),</span>
|
||||
<span class="n">sqrt</span><span class="p">(</span><span class="n">d1</span><span class="o">+</span><span class="n">d2</span><span class="p">)</span> <span class="p">)</span>
|
||||
<span class="n">d3</span> <span class="o">=</span> <span class="n">div</span><span class="p">(</span><span class="n">d2</span> <span class="o">-</span> <span class="n">d1</span><span class="p">,</span> <span class="n">d1</span> <span class="o">+</span> <span class="n">d2</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Asc = A*diag(d)^-1/2</span>
|
||||
<span class="n">Asc</span> <span class="o">=</span> <span class="n">A</span> <span class="o">*</span> <span class="n">spdiag</span><span class="p">(</span><span class="n">ds</span><span class="o">**-</span><span class="mi">1</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># S = I + A * D^-1 * A'</span>
|
||||
<span class="n">blas</span><span class="o">.</span><span class="n">syrk</span><span class="p">(</span><span class="n">Asc</span><span class="p">,</span> <span class="n">S</span><span class="p">)</span>
|
||||
<span class="n">S</span><span class="p">[::</span><span class="n">m</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="mf">1.0</span>
|
||||
<span class="n">lapack</span><span class="o">.</span><span class="n">potrf</span><span class="p">(</span><span class="n">S</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">g</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">z</span><span class="p">):</span>
|
||||
|
||||
<span class="n">x</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="mf">0.5</span> <span class="o">*</span> <span class="p">(</span> <span class="n">x</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">-</span> <span class="n">mul</span><span class="p">(</span><span class="n">d3</span><span class="p">,</span> <span class="n">x</span><span class="p">[</span><span class="n">n</span><span class="p">:])</span> <span class="o">+</span>
|
||||
<span class="n">mul</span><span class="p">(</span><span class="n">d1</span><span class="p">,</span> <span class="n">z</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">+</span> <span class="n">mul</span><span class="p">(</span><span class="n">d3</span><span class="p">,</span> <span class="n">z</span><span class="p">[:</span><span class="n">n</span><span class="p">]))</span> <span class="o">-</span> <span class="n">mul</span><span class="p">(</span><span class="n">d2</span><span class="p">,</span> <span class="n">z</span><span class="p">[</span><span class="n">n</span><span class="p">:]</span> <span class="o">-</span>
|
||||
<span class="n">mul</span><span class="p">(</span><span class="n">d3</span><span class="p">,</span> <span class="n">z</span><span class="p">[</span><span class="n">n</span><span class="p">:]))</span> <span class="p">)</span>
|
||||
<span class="n">x</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="n">div</span><span class="p">(</span> <span class="n">x</span><span class="p">[:</span><span class="n">n</span><span class="p">],</span> <span class="n">ds</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Solve</span>
|
||||
<span class="c1">#</span>
|
||||
<span class="c1"># S * v = 0.5 * A * D^-1 * ( bx[:n] - </span>
|
||||
<span class="c1"># (D2-D1)*(D1+D2)^-1 * bx[n:] + </span>
|
||||
<span class="c1"># D1 * ( I + (D2-D1)*(D1+D2)^-1 ) * bzl[:n] - </span>
|
||||
<span class="c1"># D2 * ( I - (D2-D1)*(D1+D2)^-1 ) * bzl[n:] )</span>
|
||||
|
||||
<span class="n">blas</span><span class="o">.</span><span class="n">gemv</span><span class="p">(</span><span class="n">Asc</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span>
|
||||
<span class="n">lapack</span><span class="o">.</span><span class="n">potrs</span><span class="p">(</span><span class="n">S</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># x[:n] = D^-1 * ( rhs - A'*v ).</span>
|
||||
<span class="n">blas</span><span class="o">.</span><span class="n">gemv</span><span class="p">(</span><span class="n">Asc</span><span class="p">,</span> <span class="n">v</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=-</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">beta</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">trans</span><span class="o">=</span><span class="s1">'T'</span><span class="p">)</span>
|
||||
<span class="n">x</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="n">div</span><span class="p">(</span><span class="n">x</span><span class="p">[:</span><span class="n">n</span><span class="p">],</span> <span class="n">ds</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># x[n:] = (D1+D2)^-1 * ( bx[n:] - D1*bzl[:n] - D2*bzl[n:] ) </span>
|
||||
<span class="c1"># - (D2-D1)*(D1+D2)^-1 * x[:n] </span>
|
||||
<span class="n">x</span><span class="p">[</span><span class="n">n</span><span class="p">:]</span> <span class="o">=</span> <span class="n">div</span><span class="p">(</span> <span class="n">x</span><span class="p">[</span><span class="n">n</span><span class="p">:]</span> <span class="o">-</span> <span class="n">mul</span><span class="p">(</span><span class="n">d1</span><span class="p">,</span> <span class="n">z</span><span class="p">[:</span><span class="n">n</span><span class="p">])</span> <span class="o">-</span> <span class="n">mul</span><span class="p">(</span><span class="n">d2</span><span class="p">,</span> <span class="n">z</span><span class="p">[</span><span class="n">n</span><span class="p">:]),</span> <span class="n">d1</span><span class="o">+</span><span class="n">d2</span> <span class="p">)</span>\
|
||||
<span class="o">-</span> <span class="n">mul</span><span class="p">(</span> <span class="n">d3</span><span class="p">,</span> <span class="n">x</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="p">)</span>
|
||||
|
||||
<span class="c1"># zl[:n] = D1^1/2 * ( x[:n] - x[n:] - bzl[:n] )</span>
|
||||
<span class="c1"># zl[n:] = D2^1/2 * ( -x[:n] - x[n:] - bzl[n:] ).</span>
|
||||
<span class="n">z</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="n">mul</span><span class="p">(</span> <span class="n">W</span><span class="p">[</span><span class="s1">'di'</span><span class="p">][:</span><span class="n">n</span><span class="p">],</span> <span class="n">x</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">-</span> <span class="n">x</span><span class="p">[</span><span class="n">n</span><span class="p">:]</span> <span class="o">-</span> <span class="n">z</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="p">)</span>
|
||||
<span class="n">z</span><span class="p">[</span><span class="n">n</span><span class="p">:]</span> <span class="o">=</span> <span class="n">mul</span><span class="p">(</span> <span class="n">W</span><span class="p">[</span><span class="s1">'di'</span><span class="p">][</span><span class="n">n</span><span class="p">:],</span> <span class="o">-</span><span class="n">x</span><span class="p">[:</span><span class="n">n</span><span class="p">]</span> <span class="o">-</span> <span class="n">x</span><span class="p">[</span><span class="n">n</span><span class="p">:]</span> <span class="o">-</span> <span class="n">z</span><span class="p">[</span><span class="n">n</span><span class="p">:]</span> <span class="p">)</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">g</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">solvers</span><span class="o">.</span><span class="n">coneqp</span><span class="p">(</span><span class="n">P</span><span class="p">,</span> <span class="n">q</span><span class="p">,</span> <span class="n">G</span><span class="p">,</span> <span class="n">h</span><span class="p">,</span> <span class="n">kktsolver</span> <span class="o">=</span> <span class="n">Fkkt</span><span class="p">)[</span><span class="s1">'x'</span><span class="p">][:</span><span class="n">n</span><span class="p">]</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
|
||||
<span class="ne">ModuleNotFoundError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
|
||||
<span class="o"><</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="n">d670a873ab0c</span><span class="o">></span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
|
||||
<span class="ne">----> </span><span class="mi">1</span> <span class="kn">from</span> <span class="nn">cvxopt</span> <span class="kn">import</span> <span class="n">matrix</span><span class="p">,</span> <span class="n">spdiag</span><span class="p">,</span> <span class="n">mul</span><span class="p">,</span> <span class="n">div</span><span class="p">,</span> <span class="n">sqrt</span><span class="p">,</span> <span class="n">normal</span><span class="p">,</span> <span class="n">setseed</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">2</span> <span class="kn">from</span> <span class="nn">cvxopt</span> <span class="kn">import</span> <span class="n">blas</span><span class="p">,</span> <span class="n">lapack</span><span class="p">,</span> <span class="n">solvers</span><span class="p">,</span> <span class="n">sparse</span><span class="p">,</span> <span class="n">spmatrix</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">3</span> <span class="kn">import</span> <span class="nn">math</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">4</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">5</span> <span class="k">try</span><span class="p">:</span>
|
||||
|
||||
<span class="ne">ModuleNotFoundError</span>: No module named 'cvxopt'
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Then we call the above functions and solve the problem, as done here</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cvxopt</span> <span class="kn">import</span> <span class="n">matrix</span><span class="p">,</span> <span class="n">normal</span>
|
||||
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">matrix</span><span class="p">(</span> <span class="p">[</span> <span class="p">[</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">]])</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">matrix</span><span class="p">(</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">l1regls</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="n">y</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>More text will be added to this example.</strong></p>
|
||||
<p>As a small addendum, we note that you can also solve this problem using the convex optimization package <a class="reference external" href="https://cvxopt.org/examples/mlbook/l1regls.html">CVXOPT</a>. This requires, in addition to having installed <strong>CVXOPT</strong>, you need to download the file <em><a class="reference external" href="http://l1regl.py">l1regl.py</a></em>.</p>
|
||||
</div>
|
||||
<div class="section" id="linking-the-regression-analysis-with-a-statistical-interpretation">
|
||||
<h2><span class="section-number">4.11. </span>Linking the regression analysis with a statistical interpretation<a class="headerlink" href="#linking-the-regression-analysis-with-a-statistical-interpretation" title="Permalink to this headline">¶</a></h2>
|
||||
@@ -2971,6 +2678,14 @@ order to another one.</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.0169643 0.27924636 -1.4087793 1.03308408 0. ]
|
||||
Test MSE OLS
|
||||
0.958228616652075
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter2_324_1.png" src="_images/chapter2_324_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>How can we understand this?</p>
|
||||
<p>Let us write out the values of the coefficients <span class="math notranslate nohighlight">\(\beta_i\)</span> as functions
|
||||
@@ -3032,6 +2747,228 @@ large variance (normally for higher orders in the polynomial).</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>beta</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>0.986699</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>-0.606760</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>1.280573</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>-0.850164</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>0.000000</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div><div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>beta</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>0.978553</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>-0.511888</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>1.051418</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>-0.701370</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>0.000000</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div><div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>beta</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>0.946957</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>-0.162246</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>0.221921</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>-0.167787</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>0.000000</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div><div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>beta</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>0.906747</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>0.017665</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>-0.029483</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>-0.053849</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>0.000000</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div><div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>beta</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>0.718165</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>0.156956</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>0.040102</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>-0.001880</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>0.000000</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div></div>
|
||||
</div>
|
||||
<p>As an exercise, repeat these calculations with ordinary least squares
|
||||
only with and without noise. Calculate thereafter the variance of the
|
||||
@@ -3074,8 +3011,7 @@ already modeled and an unknown prior, we are now ready to make
|
||||
additional models for the prior.</p>
|
||||
<p>We can, based on our discussions of the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and
|
||||
the mean value, assume that the prior for the values <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is
|
||||
given by a Gaussian with mean value zero and variance <span class="math notranslate nohighlight">\(\tau^2\)</span>, that
|
||||
is</p>
|
||||
given by a Gaussian with mean value zero and variance <span class="math notranslate nohighlight">\(\tau^2\)</span>, that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}.
|
||||
|
||||
Reference in New Issue
Block a user