update
This commit is contained in:
Morten Hjorth-Jensen
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 18 KiB After Width: | Height: | Size: 18 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 25 KiB After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 24 KiB After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 21 KiB After Width: | Height: | Size: 21 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 22 KiB After Width: | Height: | Size: 21 KiB |
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -644,7 +644,7 @@ role when we develop a specific machine learning algorithm.</p>
|
||||
<p>Machine learning is an extremely rich field, in spite of its young
|
||||
age. The increases we have seen during the last three decades in
|
||||
computational capabilities have been followed by developments of
|
||||
methods and techniques for analyzing and handling large date sets,
|
||||
methods and techniques for analyzing and handling large data sets,
|
||||
relying heavily on statistics, computer science and mathematics. The
|
||||
field is rather new and developing rapidly. Popular software packages
|
||||
written in Python for machine learning like
|
||||
@@ -666,7 +666,7 @@ two main categories. In <em>supervised learning</em> we know the answer to a
|
||||
problem, and let the computer deduce the logic behind it. On the other
|
||||
hand, <em>unsupervised learning</em> is a method for finding patterns and
|
||||
relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely
|
||||
Some authors also operate with a third category, namely
|
||||
<em>reinforcement learning</em>. This is a paradigm of learning inspired by
|
||||
behavioral psychology, where learning is achieved by trial-and-error,
|
||||
solely from rewards and punishment.</p>
|
||||
@@ -714,13 +714,13 @@ what is the likelihood of finding <span class="math notranslate nohighlight">\(B
|
||||
<h3><span class="section-number">3.2.2. </span>What is a good model?<a class="headerlink" href="#what-is-a-good-model" title="Permalink to this headline">¶</a></h3>
|
||||
<p>In science and engineering we often end up in situations where we want to infer (or learn) a
|
||||
quantitative model <span class="math notranslate nohighlight">\(M\)</span> for a given set of sample points <span class="math notranslate nohighlight">\(\boldsymbol{X} \in [x_1, x_2,\dots x_N]\)</span>.</p>
|
||||
<p>As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a
|
||||
<p>As we will see repeatedly in these lectures, we could try to fit these data points to a model given by a
|
||||
straight line, or if we wish to be more sophisticated to a more complex
|
||||
function.</p>
|
||||
<p>The reason for inferring such a model is that it
|
||||
serves many useful purposes. On the one hand, the model can reveal information
|
||||
encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important
|
||||
corelations that relate interesting physics interpretations.</p>
|
||||
correlations that relate interesting physics interpretations.</p>
|
||||
<p>In addition, it can simplify the representation of the given data set and help
|
||||
us in making predictions about future data samples.</p>
|
||||
<p>A first important consideration to keep in mind is that inferring the <em>correct</em> model
|
||||
@@ -843,7 +843,7 @@ y = 10x+0.01 \times N(0,1),
|
||||
<p>where <span class="math notranslate nohighlight">\(x\)</span> is defined as before. Does the fit look better? Indeed, by
|
||||
reducing the role of the noise given by the normal distribution we see immediately that
|
||||
our linear prediction seemingly reproduces better the training
|
||||
set. However, this testing ‘by the eye’ is obviouly not satisfactory in the
|
||||
set. However, this testing ‘by the eye’ is obviously not satisfactory in the
|
||||
long run. Here we have only defined the training data and our model, and
|
||||
have not discussed a more rigorous approach to the <strong>cost</strong> function.</p>
|
||||
<p>We need more rigorous criteria in defining whether we have succeeded or
|
||||
@@ -957,13 +957,13 @@ example of the functionality of <strong>Scikit-Learn</strong>.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The intercept alpha:
|
||||
[2.02408959]
|
||||
[2.03523311]
|
||||
Coefficient beta :
|
||||
[[4.92811987]]
|
||||
Mean squared error: 0.25
|
||||
Variance score: 0.89
|
||||
[[4.99498108]]
|
||||
Mean squared error: 0.27
|
||||
Variance score: 0.87
|
||||
Mean squared log error: 0.01
|
||||
Mean absolute error: 0.39
|
||||
Mean absolute error: 0.41
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_19_1.png" src="_images/chapter1_19_1.png" />
|
||||
@@ -1063,7 +1063,7 @@ a linear <span class="math notranslate nohighlight">\(x\)</span>-dependence we s
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_33_0.png" src="_images/chapter1_33_0.png" />
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.005
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.00499999999999999
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -551,7 +551,7 @@ later shrinkage methods like Ridge and Lasso regressions.</p>
|
||||
<p>This is given by the <strong>Singular Value Decomposition</strong> (SVD) algorithm,
|
||||
perhaps the most powerful linear algebra algorithm. The SVD provides
|
||||
a numerically stable matrix decomposition that is used in a large
|
||||
swath oc applications and the decomposition is always stable
|
||||
swath of applications and the decomposition is always stable
|
||||
numerically.</p>
|
||||
<p>In machine learning it plays a central role in dealing with for
|
||||
example design matrices that may be near singular or singular.
|
||||
@@ -565,7 +565,7 @@ when the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</s
|
||||
are problems with near singular or singular matrices. The column vectors of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>
|
||||
may be linearly dependent, normally referred to as super-collinearity.<br />
|
||||
This means that the matrix may be rank deficient and it is basically impossible to
|
||||
to model the data using linear regression. As an example, consider the matrix</p>
|
||||
model the data using linear regression. As an example, consider the matrix</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\begin{align*}
|
||||
@@ -585,7 +585,7 @@ to model the data using linear regression. As an example, consider the matrix</p
|
||||
the first column is the row-wise sum of the other two columns. The rank (more correct,
|
||||
the column rank) of a matrix is the dimension of the space spanned by the
|
||||
column vectors. Hence, the rank of <span class="math notranslate nohighlight">\(\mathbf{X}\)</span> is equal to the number
|
||||
of linearly independent columns. In this particular case the matrix has rank 2.</p>
|
||||
of linearly independent columns. In this particular case the matrix has rank 1.</p>
|
||||
<p>Super-collinearity of an <span class="math notranslate nohighlight">\((n \times p)\)</span>-dimensional design matrix <span class="math notranslate nohighlight">\(\mathbf{X}\)</span> implies
|
||||
that the inverse of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -613,7 +613,7 @@ This is equivalent to saying that the matrix <span class="math notranslate nohig
|
||||
\]</div>
|
||||
<p>has linearly dependent column vectors, we will not be able to compute the inverse
|
||||
of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> and we cannot find the parameters (estimators) <span class="math notranslate nohighlight">\(\beta_i\)</span>.
|
||||
The estimators are only well-defined if <span class="math notranslate nohighlight">\((\boldsymbol{X}^{T}\boldsymbol{X})^{-1}\)</span> exits.
|
||||
The estimators are only well-defined if <span class="math notranslate nohighlight">\((\boldsymbol{X}^{T}\boldsymbol{X})\)</span> can be inverted.
|
||||
This is more likely to happen when the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is high-dimensional. In this case it is likely to encounter a situation where
|
||||
the regression parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> cannot be estimated.</p>
|
||||
<p>A cheap <em>ad hoc</em> approach is simply to add a small diagonal component to the matrix to invert, that is we change</p>
|
||||
@@ -625,7 +625,7 @@ the regression parameters <span class="math notranslate nohighlight">\(\beta_i\)
|
||||
</div>
|
||||
<div class="section" id="basic-math-of-the-svd">
|
||||
<h2><span class="section-number">4.3. </span>Basic math of the SVD<a class="headerlink" href="#basic-math-of-the-svd" title="Permalink to this headline">¶</a></h2>
|
||||
<p>From standard linear algebra we know that a square matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> can be diagonalized if and only it is
|
||||
<p>From standard linear algebra we know that a square matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> can be diagonalized if and only if it is
|
||||
a so-called <a class="reference external" href="https://en.wikipedia.org/wiki/Normal_matrix">normal matrix</a>, that is if <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times n}\)</span>
|
||||
we have <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X}\)</span> or if <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{C}}^{n\times n}\)</span> we have <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X}\)</span>.
|
||||
The matrix has then a set of eigenpairs</p>
|
||||
@@ -813,7 +813,6 @@ The simple answer is to use the linear algebra function for the pseudoinverse, t
|
||||
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">V</span><span class="p">,</span><span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">invD</span><span class="p">,</span><span class="n">UT</span><span class="p">))</span>
|
||||
|
||||
|
||||
<span class="c1">#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])</span>
|
||||
<span class="c1"># Non-singular square matrix</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span> <span class="p">[</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">],[</span><span class="mi">2</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">],[</span><span class="mi">3</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">]])</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
|
||||
@@ -849,7 +848,7 @@ test VT
|
||||
rectangular matrices where the number of rows and columns are not equal.</p>
|
||||
<p>It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.
|
||||
It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.</p>
|
||||
<p>Using the SVD we can obtain the pseudoinverse of a matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> (labeled here as <span class="math notranslate nohighlight">\(\boldsymbol{A}_{\mathrm{PI}}\)</span></p>
|
||||
<p>Using the SVD we can obtain the pseudoinverse (PI) of a matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> (labeled here as <span class="math notranslate nohighlight">\(\boldsymbol{A}_{\mathrm{PI}}\)</span></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T,
|
||||
@@ -919,7 +918,7 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
|
||||
\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
|
||||
\]</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> is an orthogonal matrix of dimension <span class="math notranslate nohighlight">\(n\times n\)</span>, meaning that <span class="math notranslate nohighlight">\(\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n\)</span>. Here <span class="math notranslate nohighlight">\(\boldsymbol{I}_n\)</span> is the unit matrix of dimension <span class="math notranslate nohighlight">\(n \times n\)</span>.</p>
|
||||
<p>Similarly, <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> is an orthogonal matrix of dimension <span class="math notranslate nohighlight">\(p\times p\)</span>, meaning that <span class="math notranslate nohighlight">\(\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p\)</span>. Here <span class="math notranslate nohighlight">\(\boldsymbol{I}_n\)</span> is the unit matrix of dimension <span class="math notranslate nohighlight">\(p \times p\)</span>.</p>
|
||||
<p>Similarly, <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> is an orthogonal matrix of dimension <span class="math notranslate nohighlight">\(p\times p\)</span>, meaning that <span class="math notranslate nohighlight">\(\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p\)</span>. Here <span class="math notranslate nohighlight">\(\boldsymbol{I}_p\)</span> is the unit matrix of dimension <span class="math notranslate nohighlight">\(p \times p\)</span>.</p>
|
||||
<p>Finally <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> contains the singular values <span class="math notranslate nohighlight">\(\sigma_i\)</span>. This matrix has dimension <span class="math notranslate nohighlight">\(n\times p\)</span> and the singular values <span class="math notranslate nohighlight">\(\sigma_i\)</span> are all positive. The non-zero values are ordered in descending order, that is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
@@ -1057,7 +1056,7 @@ function, that is we have</p>
|
||||
\[
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\]</div>
|
||||
<p>This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).</p>
|
||||
<p>This quantity defines what is called the Hessian matrix (the second derivative of the cost function we want to optimize).</p>
|
||||
<p>The Hessian matrix plays an important role and is defined in this course as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
@@ -1148,7 +1147,7 @@ We can rewrite the design/feature matrix in terms of its column vectors as</p>
|
||||
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
|
||||
\]</div>
|
||||
<p>With these definitions, we can now rewrite our <span class="math notranslate nohighlight">\(2\times 2\)</span>
|
||||
correlation/covariance matrix in terms of a moe general design/feature
|
||||
correlation/covariance matrix in terms of a more general design/feature
|
||||
matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>. This leads to a <span class="math notranslate nohighlight">\(p\times p\)</span>
|
||||
covariance matrix for the vectors <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> with <span class="math notranslate nohighlight">\(i=0,1,\dots,p-1\)</span></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -1207,10 +1206,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.04413933503955871
|
||||
4.12330280229368
|
||||
[[0.80162359 2.38222896]
|
||||
[2.38222896 8.12167821]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.08873443359350565
|
||||
3.7851533175757255
|
||||
[[ 0.98248312 3.05483267]
|
||||
[ 3.05483267 10.24784064]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1247,10 +1246,10 @@ a more brute force way. Here we scale the mean values for each column of the des
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.06786925114666595
|
||||
1.9635449873404844
|
||||
[[1. 0.65522261]
|
||||
[0.65522261 1. ]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07858099596662704
|
||||
2.071920625289855
|
||||
[[1. 0.71822416]
|
||||
[0.71822416 1. ]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1259,7 +1258,7 @@ a more brute force way. Here we scale the mean values for each column of the des
|
||||
should be and that the matrix is symmetric. Furthermore, diagonalizing
|
||||
this matrix we easily see that it is a positive definite matrix.</p>
|
||||
<p>The above procedure with <strong>numpy</strong> can be made more compact if we use <strong>pandas</strong>.</p>
|
||||
<p>We whow here how we can set up the correlation matrix using <strong>pandas</strong>, as done in this simple code</p>
|
||||
<p>We know here how we can set up the correlation matrix using <strong>pandas</strong>, as done in this simple code</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
@@ -1280,30 +1279,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-0.27091656 -1.29083183]
|
||||
[ 0.31980301 0.87495119]
|
||||
[-0.10835935 1.61413333]
|
||||
[ 0.5188328 2.80380438]
|
||||
[-0.04996008 -1.95742107]
|
||||
[ 1.19432526 2.68719389]
|
||||
[ 0.19710439 1.35590603]
|
||||
[-0.23857423 -2.50104946]
|
||||
[-0.94054854 -2.09034902]
|
||||
[-0.62170669 -1.49633743]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ -2.84861838 -10.07337358]
|
||||
[ 0.53938383 2.59445979]
|
||||
[ -0.40980089 -0.48871288]
|
||||
[ 0.05834332 -0.39384255]
|
||||
[ 2.25385387 7.58112299]
|
||||
[ 0.68246434 2.46650488]
|
||||
[ -0.25366775 -1.97047717]
|
||||
[ 0.79081838 2.03807267]
|
||||
[ -0.06150169 -0.57109235]
|
||||
[ -0.75127504 -1.18266178]]
|
||||
0 1
|
||||
0 -2.848618 -10.073374
|
||||
1 0.539384 2.594460
|
||||
2 -0.409801 -0.488713
|
||||
3 0.058343 -0.393843
|
||||
4 2.253854 7.581123
|
||||
5 0.682464 2.466505
|
||||
6 -0.253668 -1.970477
|
||||
7 0.790818 2.038073
|
||||
8 -0.061502 -0.571092
|
||||
9 -0.751275 -1.182662
|
||||
0 1
|
||||
0 -0.270917 -1.290832
|
||||
1 0.319803 0.874951
|
||||
2 -0.108359 1.614133
|
||||
3 0.518833 2.803804
|
||||
4 -0.049960 -1.957421
|
||||
5 1.194325 2.687194
|
||||
6 0.197104 1.355906
|
||||
7 -0.238574 -2.501049
|
||||
8 -0.940549 -2.090349
|
||||
9 -0.621707 -1.496337
|
||||
0 1
|
||||
0 1.000000 0.800615
|
||||
1 0.800615 1.000000
|
||||
0 1.000000 0.984525
|
||||
1 0.984525 1.000000
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1360,37 +1359,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
|
||||
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
|
||||
1 0.0 0.084006 0.079882 0.084682 0.084092 0.083417 0.076315 0.076097
|
||||
2 0.0 0.079882 0.077644 0.078542 0.078962 0.079424 0.069534 0.069977
|
||||
3 0.0 0.084682 0.078542 0.090649 0.088758 0.086665 0.085105 0.084008
|
||||
4 0.0 0.084092 0.078962 0.088758 0.087573 0.086250 0.082424 0.081832
|
||||
5 0.0 0.083417 0.079424 0.086665 0.086250 0.085776 0.079486 0.079438
|
||||
6 0.0 0.076315 0.069534 0.085105 0.082424 0.079486 0.082288 0.080588
|
||||
7 0.0 0.076097 0.069977 0.084008 0.081832 0.079438 0.080588 0.079264
|
||||
8 0.0 0.076022 0.070618 0.082990 0.081357 0.079553 0.078908 0.077986
|
||||
9 0.0 0.076079 0.071460 0.082027 0.080984 0.079823 0.077219 0.076729
|
||||
10 0.0 0.068075 0.061188 0.078143 0.075043 0.071666 0.077200 0.075149
|
||||
11 0.0 0.067712 0.061308 0.077144 0.074420 0.071445 0.075770 0.074006
|
||||
12 0.0 0.067499 0.061604 0.076264 0.073938 0.071388 0.074418 0.072955
|
||||
13 0.0 0.067443 0.062089 0.075498 0.073597 0.071505 0.073134 0.071991
|
||||
14 0.0 0.067547 0.062777 0.074845 0.073400 0.071804 0.071908 0.071106
|
||||
1 0.0 0.090241 0.082140 0.090564 0.084086 0.078082 0.082282 0.076619
|
||||
2 0.0 0.082140 0.075227 0.083102 0.077428 0.072150 0.075982 0.070945
|
||||
3 0.0 0.090564 0.083102 0.096893 0.090268 0.084107 0.091647 0.085571
|
||||
4 0.0 0.084086 0.077428 0.090268 0.084286 0.078707 0.085655 0.080120
|
||||
5 0.0 0.078082 0.072150 0.084107 0.078707 0.073657 0.080061 0.075020
|
||||
6 0.0 0.082282 0.075982 0.091647 0.085655 0.080061 0.089082 0.083380
|
||||
7 0.0 0.076619 0.070945 0.085571 0.080120 0.075020 0.083380 0.078158
|
||||
8 0.0 0.071394 0.066284 0.079944 0.074984 0.070333 0.078082 0.073299
|
||||
9 0.0 0.066569 0.061966 0.074729 0.070216 0.065973 0.073159 0.068776
|
||||
10 0.0 0.073831 0.068541 0.084523 0.079224 0.074258 0.083779 0.078587
|
||||
11 0.0 0.068867 0.064081 0.079021 0.074183 0.069640 0.078484 0.073716
|
||||
12 0.0 0.064284 0.059952 0.073925 0.069506 0.065349 0.073567 0.069187
|
||||
13 0.0 0.060048 0.056127 0.069202 0.065165 0.061359 0.068999 0.064974
|
||||
14 0.0 0.056131 0.052581 0.064823 0.061133 0.057648 0.064753 0.061054
|
||||
|
||||
8 9 10 11 12 13 14
|
||||
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
|
||||
1 0.076022 0.076079 0.068075 0.067712 0.067499 0.067443 0.067547
|
||||
2 0.070618 0.071460 0.061188 0.061308 0.061604 0.062089 0.062777
|
||||
3 0.082990 0.082027 0.078143 0.077144 0.076264 0.075498 0.074845
|
||||
4 0.081357 0.080984 0.075043 0.074420 0.073938 0.073597 0.073400
|
||||
5 0.079553 0.079823 0.071666 0.071445 0.071388 0.071505 0.071804
|
||||
6 0.078908 0.077219 0.077200 0.075770 0.074418 0.073134 0.071908
|
||||
7 0.077986 0.076729 0.075149 0.074006 0.072955 0.071991 0.071106
|
||||
8 0.077140 0.076349 0.073080 0.072240 0.071510 0.070887 0.070370
|
||||
9 0.076349 0.076066 0.070961 0.070443 0.070056 0.069801 0.069681
|
||||
10 0.073080 0.070961 0.073601 0.071922 0.070288 0.068689 0.067110
|
||||
11 0.072240 0.070443 0.071922 0.070466 0.069065 0.067709 0.066388
|
||||
12 0.071510 0.070056 0.070288 0.069065 0.067907 0.066808 0.065761
|
||||
13 0.070887 0.069801 0.068689 0.067709 0.066808 0.065983 0.065228
|
||||
14 0.070370 0.069681 0.067110 0.066388 0.065761 0.065228 0.064787
|
||||
1 0.071394 0.066569 0.073831 0.068867 0.064284 0.060048 0.056131
|
||||
2 0.066284 0.061966 0.068541 0.064081 0.059952 0.056127 0.052581
|
||||
3 0.079944 0.074729 0.084523 0.079021 0.073925 0.069202 0.064823
|
||||
4 0.074984 0.070216 0.079224 0.074183 0.069506 0.065165 0.061133
|
||||
5 0.070333 0.065973 0.074258 0.069640 0.065349 0.061359 0.057648
|
||||
6 0.078082 0.073159 0.083779 0.078484 0.073567 0.068999 0.064753
|
||||
7 0.073299 0.068776 0.078587 0.073716 0.069187 0.064974 0.061054
|
||||
8 0.068841 0.064684 0.073750 0.069268 0.065095 0.061209 0.057588
|
||||
9 0.064684 0.060863 0.069242 0.065118 0.061272 0.057686 0.054340
|
||||
10 0.073750 0.069242 0.079948 0.075028 0.070450 0.066189 0.062220
|
||||
11 0.069268 0.065118 0.075028 0.070494 0.066270 0.062333 0.058663
|
||||
12 0.065095 0.061272 0.070450 0.066270 0.062370 0.058732 0.055337
|
||||
13 0.061209 0.057686 0.066189 0.062333 0.058732 0.055369 0.052227
|
||||
14 0.057588 0.054340 0.062220 0.058663 0.055337 0.052227 0.049318
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1668,7 +1667,7 @@ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\bolds
|
||||
<p>This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a class="reference external" href="https://cvxopt.org/">CVXOPT</a>. We will discuss this later.</p>
|
||||
<p>Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
|
||||
diagonal. In this case we have an equal number of rows and columns <span class="math notranslate nohighlight">\(n=p\)</span>.</p>
|
||||
<p>Our model approximation is just <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}=\boldsymbol{\beta}\)</span> and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term <span class="math notranslate nohighlight">\(1/n\)</span>)</p>
|
||||
<p>Our model approximation is just <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}=\boldsymbol{\beta}\)</span> and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term <span class="math notranslate nohighlight">\(1/n\)</span>)</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
|
||||
@@ -1706,7 +1705,7 @@ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\ve
|
||||
0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\.
|
||||
\end{split}\]</div>
|
||||
<p>Plotting these results (<a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf">figure in handwritten notes for week 36</a>) shows clearly that Lasso regression suppresses (sets to zero) values of <span class="math notranslate nohighlight">\(\beta_i\)</span> for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>. Ridge regression reduces on the other hand the values of <span class="math notranslate nohighlight">\(\beta_i\)</span> as function of <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
|
||||
<p>As another examples,
|
||||
<p>As another example,
|
||||
let us assume we have a data set with outputs/targets given by the vector</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
@@ -1878,12 +1877,12 @@ Training MSE for OLS
|
||||
<img alt="_images/chapter2_252_1.png" src="_images/chapter2_252_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
|
||||
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
|
||||
<p>This happens also for Lasso regression, as seen from the next code
|
||||
output. The difference is that Lasso shrinks the values of <span class="math notranslate nohighlight">\(\beta\)</span> to
|
||||
zero at a much earlier stage and the results flatten out. We see that
|
||||
Lasso gives also an excellent fit for small values of <span class="math notranslate nohighlight">\(\lambda\)</span> and
|
||||
shows rthe best performance of the three regression methods.</p>
|
||||
shows the best performance of the three regression methods.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
@@ -2426,7 +2425,7 @@ We define this distribution as</p>
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
|
||||
\]</div>
|
||||
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
|
||||
<p>Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as the product of the single events, that is we have</p>
|
||||
<p>Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as the product of the single events, that is we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
|
||||
@@ -2497,7 +2496,7 @@ p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
|
||||
<p>The product rule (aka joint probability) is given by</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
|
||||
p(X \cap Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
|
||||
\]</div>
|
||||
<p>where we read <span class="math notranslate nohighlight">\(p(X\vert Y)\)</span> as the likelihood of obtaining <span class="math notranslate nohighlight">\(X\)</span> given <span class="math notranslate nohighlight">\(Y\)</span>.</p>
|
||||
<p>If we have independent events then <span class="math notranslate nohighlight">\(p(X,Y)=p(X)p(Y)\)</span>.</p>
|
||||
@@ -2973,7 +2972,7 @@ parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> as func
|
||||
noise. Here we recommend to use <span class="math notranslate nohighlight">\(\sigma^2=1\)</span> as variance for the
|
||||
added noise (which follows a normal distribution with mean value zero).
|
||||
Comment your results. If you have a large noise term, do the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> vary more as function
|
||||
model complexity? And what about their variance?</p>
|
||||
of model complexity? And what about their variance?</p>
|
||||
</div>
|
||||
<div class="section" id="linking-bayes-theorem-with-ridge-and-lasso-regression">
|
||||
<h2><span class="section-number">4.14. </span>Linking Bayes’ Theorem with Ridge and Lasso Regression<a class="headerlink" href="#linking-bayes-theorem-with-ridge-and-lasso-regression" title="Permalink to this headline">¶</a></h2>
|
||||
@@ -3002,7 +3001,7 @@ p(\boldsymbol{\beta}\vert\boldsymbol{D}).
|
||||
\[
|
||||
p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}).
|
||||
\]</div>
|
||||
<p>We have a model for <span class="math notranslate nohighlight">\(p(\boldsymbol{D}\vert\boldsymbol{\beta})\)</span> but need one for the <strong>prior</strong> <span class="math notranslate nohighlight">\(p(\boldsymbol{\beta}\)</span>!</p>
|
||||
<p>We have a model for <span class="math notranslate nohighlight">\(p(\boldsymbol{D}\vert\boldsymbol{\beta})\)</span> but need one for the <strong>prior</strong> <span class="math notranslate nohighlight">\(p(\boldsymbol{\beta})\)</span>!</p>
|
||||
<p>With the posterior probability defined by a likelihood which we have
|
||||
already modeled and an unknown prior, we are now ready to make
|
||||
additional models for the prior.</p>
|
||||
@@ -3047,12 +3046,12 @@ logarithm of the posterior probability and leaving out the
|
||||
constants terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
\]</div>
|
||||
<p>and replacing <span class="math notranslate nohighlight">\(1/\tau\)</span> with <span class="math notranslate nohighlight">\(\lambda\)</span> we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
\]</div>
|
||||
<p>which is our Lasso cost function!</p>
|
||||
<p>Plotting these prior functions shows us that we can use the parameter
|
||||
|
||||
@@ -389,22 +389,37 @@ const thebe_selector_output = ".output, .cell_output"
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#using-autograd-with-ols">
|
||||
7.11. Using Autograd with OLS
|
||||
<a class="reference internal nav-link" href="#replace-or-not">
|
||||
7.11. Replace or not
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#using-autograd">
|
||||
7.12. Using Autograd
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#same-code-but-now-with-momentum-gradient-descent">
|
||||
7.13. Same code but now with momentum gradient descent
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">
|
||||
7.14. Including Stochastic Gradient Descent with Autograd
|
||||
</a>
|
||||
<ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">
|
||||
7.11.1. Including Stochastic Gradient Descent with Autograd
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#and-logistic-regression">
|
||||
7.11.2. And Logistic Regression
|
||||
<a class="reference internal nav-link" href="#similar-second-order-function-now-problem-but-now-with-adagrad">
|
||||
7.14.1. Similar (second order function now) problem but now with AdaGrad
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#introducing-jax">
|
||||
7.15. Introducing JAX
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -502,22 +517,37 @@ const thebe_selector_output = ".output, .cell_output"
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#using-autograd-with-ols">
|
||||
7.11. Using Autograd with OLS
|
||||
<a class="reference internal nav-link" href="#replace-or-not">
|
||||
7.11. Replace or not
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#using-autograd">
|
||||
7.12. Using Autograd
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#same-code-but-now-with-momentum-gradient-descent">
|
||||
7.13. Same code but now with momentum gradient descent
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">
|
||||
7.14. Including Stochastic Gradient Descent with Autograd
|
||||
</a>
|
||||
<ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">
|
||||
7.11.1. Including Stochastic Gradient Descent with Autograd
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#and-logistic-regression">
|
||||
7.11.2. And Logistic Regression
|
||||
<a class="reference internal nav-link" href="#similar-second-order-function-now-problem-but-now-with-adagrad">
|
||||
7.14.1. Similar (second order function now) problem but now with AdaGrad
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#introducing-jax">
|
||||
7.15. Introducing JAX
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -735,7 +765,18 @@ connecting <span class="math notranslate nohighlight">\(x\)</span> and <span cla
|
||||
<p>The convex subsets of <span class="math notranslate nohighlight">\(\mathbb{R}\)</span> are the intervals of
|
||||
<span class="math notranslate nohighlight">\(\mathbb{R}\)</span>. Examples of convex sets of <span class="math notranslate nohighlight">\(\mathbb{R}^2\)</span> are the
|
||||
regular polygons (triangles, rectangles, pentagons, etc…).</p>
|
||||
<p><strong>Convex function</strong>: Let <span class="math notranslate nohighlight">\(X \subset \mathbb{R}^n\)</span> be a convex set. Assume that the function <span class="math notranslate nohighlight">\(f: X \rightarrow \mathbb{R}\)</span> is continuous, then <span class="math notranslate nohighlight">\(f\)</span> is said to be convex if $<span class="math notranslate nohighlight">\(f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) \)</span><span class="math notranslate nohighlight">\( for all \)</span>x_1, x_2 \in X<span class="math notranslate nohighlight">\( and for all \)</span>t \in [0,1]<span class="math notranslate nohighlight">\(. If \)</span>\leq<span class="math notranslate nohighlight">\( is replaced with a strict inequaltiy in the definition, we demand \)</span>x_1 \neq x_2<span class="math notranslate nohighlight">\( and \)</span>t\in(0,1)<span class="math notranslate nohighlight">\( then \)</span>f<span class="math notranslate nohighlight">\( is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \)</span>f(x_1)<span class="math notranslate nohighlight">\( and \)</span>f(x_2)<span class="math notranslate nohighlight">\(, the value of the function on the interval \)</span>[x_1,x_2]$ is always below the line as illustrated below.</p>
|
||||
<p><strong>Convex function</strong>: Let <span class="math notranslate nohighlight">\(X \subset \mathbb{R}^n\)</span> be a convex
|
||||
set. Assume that the function <span class="math notranslate nohighlight">\(f: X \rightarrow \mathbb{R}\)</span> is
|
||||
continuous, then <span class="math notranslate nohighlight">\(f\)</span> is said to be convex if
|
||||
<span class="math notranslate nohighlight">\(f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2)\)</span>
|
||||
for all
|
||||
<span class="math notranslate nohighlight">\(x_1, x_2 \in X\)</span> and for all <span class="math notranslate nohighlight">\(t \in [0,1]\)</span>.</p>
|
||||
<p>If <span class="math notranslate nohighlight">\(\leq\)</span> is replaced with a strict inequality in the
|
||||
definition, we demand <span class="math notranslate nohighlight">\(x_1 \neq x_2\)</span> and <span class="math notranslate nohighlight">\(t\in(0,1)\)</span> then <span class="math notranslate nohighlight">\(f\)</span> is said
|
||||
to be strictly convex. For a single variable function, convexity means
|
||||
that if you draw a straight line connecting <span class="math notranslate nohighlight">\(f(x_1)\)</span> and <span class="math notranslate nohighlight">\(f(x_2)\)</span>, the
|
||||
value of the function on the interval <span class="math notranslate nohighlight">\([x_1,x_2]\)</span> is always below the
|
||||
line as discussed below.</p>
|
||||
<p>In the following we state first and second-order conditions which
|
||||
ensures convexity of a function <span class="math notranslate nohighlight">\(f\)</span>. We write <span class="math notranslate nohighlight">\(D_f\)</span> to denote the
|
||||
domain of <span class="math notranslate nohighlight">\(f\)</span>, i.e the subset of <span class="math notranslate nohighlight">\(R^n\)</span> where <span class="math notranslate nohighlight">\(f\)</span> is defined. For more
|
||||
@@ -746,7 +787,7 @@ all <span class="math notranslate nohighlight">\(x\)</span> in the domain of <sp
|
||||
is a convex set and $<span class="math notranslate nohighlight">\(f(y) \geq f(x) + \nabla f(x)^T (y-x) \)</span><span class="math notranslate nohighlight">\( holds
|
||||
for all \)</span>x,y \in D_f<span class="math notranslate nohighlight">\(. This condition means that for a convex function
|
||||
the first order Taylor expansion (right hand side above) at any point
|
||||
a global under estimator of the function. To convince yourself you can
|
||||
is a global under estimator of the function. To convince yourself you can
|
||||
make a drawing of \)</span>f(x) = x^2+1<span class="math notranslate nohighlight">\( and draw the tangent line to \)</span>f(x)$ and
|
||||
note that it is always below the graph.</p>
|
||||
<p><strong>Second order condition.</strong></p>
|
||||
@@ -899,11 +940,11 @@ which equals</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94582/483257001.py:18: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_96694/483257001.py:18: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
|
||||
ax = fig.gca(projection="3d")
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x128ee8850>
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x11db14850>
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapteroptimization_61_2.png" src="_images/chapteroptimization_61_2.png" />
|
||||
@@ -961,7 +1002,7 @@ which equals</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[<matplotlib.lines.Line2D at 0x12946f2e0>]
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[<matplotlib.lines.Line2D at 0x11e09b370>]
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapteroptimization_69_1.png" src="_images/chapteroptimization_69_1.png" />
|
||||
@@ -1218,11 +1259,11 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.34158665 3.94915262]
|
||||
[[3.97117751]
|
||||
[3.11850274]]
|
||||
[[3.97117751]
|
||||
[3.11850274]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.2831603 4.55553537]
|
||||
[[3.91511388]
|
||||
[3.13030182]]
|
||||
[[3.91511388]
|
||||
[3.13030182]]
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapteroptimization_123_1.png" src="_images/chapteroptimization_123_1.png" />
|
||||
@@ -1251,9 +1292,9 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.40754621]
|
||||
[2.78752269]]
|
||||
[4.37713991] [2.77711437]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.1509778 ]
|
||||
[2.92461411]]
|
||||
[4.13288373] [2.92817032]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1324,10 +1365,10 @@ C_{\text{ridge}}(\beta) = \frac{1}{n}||X\beta -\mathbf{y}||^2 + \lambda ||\beta|
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[3.94107596]
|
||||
[2.96620033]]
|
||||
[[3.96670977]
|
||||
[2.94212937]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.0795449 ]
|
||||
[2.86893619]]
|
||||
[[4.04785727]
|
||||
[2.89298533]]
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapteroptimization_132_1.png" src="_images/chapteroptimization_132_1.png" />
|
||||
@@ -1577,15 +1618,15 @@ function.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
|
||||
[[3.95446837]
|
||||
[3.16961682]]
|
||||
Eigenvalues of Hessian Matrix:[0.31447174 4.32459186]
|
||||
[[4.41170104]
|
||||
[2.6431453 ]]
|
||||
Eigenvalues of Hessian Matrix:[0.31228042 4.55571665]
|
||||
theta from own gd
|
||||
[[3.95446837]
|
||||
[3.16961682]]
|
||||
[[4.41170104]
|
||||
[2.6431453 ]]
|
||||
theta from own sdg
|
||||
[[3.91682433]
|
||||
[3.13655438]]
|
||||
[[4.39272691]
|
||||
[2.63430285]]
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapteroptimization_148_1.png" src="_images/chapteroptimization_148_1.png" />
|
||||
@@ -2302,8 +2343,15 @@ which also computed the dot product can be used:</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="using-autograd-with-ols">
|
||||
<h2><span class="section-number">7.11. </span>Using Autograd with OLS<a class="headerlink" href="#using-autograd-with-ols" title="Permalink to this headline">¶</a></h2>
|
||||
<div class="section" id="replace-or-not">
|
||||
<h2><span class="section-number">7.11. </span>Replace or not<a class="headerlink" href="#replace-or-not" title="Permalink to this headline">¶</a></h2>
|
||||
<p>In the above code, we have use replacement in setting up the
|
||||
mini-batches. The discussion
|
||||
<a class="reference external" href="https://sebastianraschka.com/faq/docs/sgd-methods.html">here</a> may be
|
||||
useful.</p>
|
||||
</div>
|
||||
<div class="section" id="using-autograd">
|
||||
<h2><span class="section-number">7.12. </span>Using Autograd<a class="headerlink" href="#using-autograd" title="Permalink to this headline">¶</a></h2>
|
||||
<p>We conclude the part on optmization by showing how we can make codes
|
||||
for linear regression and logistic regression using <strong>autograd</strong>. The
|
||||
first example shows results with ordinary leats squares.</p>
|
||||
@@ -2362,8 +2410,117 @@ first example shows results with ordinary leats squares.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="same-code-but-now-with-momentum-gradient-descent">
|
||||
<h2><span class="section-number">7.13. </span>Same code but now with momentum gradient descent<a class="headerlink" href="#same-code-but-now-with-momentum-gradient-descent" title="Permalink to this headline">¶</a></h2>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients for OLS</span>
|
||||
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">beta</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">beta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
|
||||
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="c1">#+np.random.randn(n,1)</span>
|
||||
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
|
||||
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
|
||||
<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Own inversion"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
|
||||
<span class="c1"># Hessian matrix</span>
|
||||
<span class="n">H</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span> <span class="n">XT_X</span>
|
||||
<span class="n">EigValues</span><span class="p">,</span> <span class="n">EigVectors</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">eig</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Eigenvalues of Hessian Matrix:</span><span class="si">{</span><span class="n">EigValues</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
|
||||
|
||||
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">eta</span> <span class="o">=</span> <span class="mf">1.0</span><span class="o">/</span><span class="n">np</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">EigValues</span><span class="p">)</span>
|
||||
<span class="n">Niterations</span> <span class="o">=</span> <span class="mi">30</span>
|
||||
|
||||
<span class="c1"># define the gradient</span>
|
||||
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">)</span>
|
||||
|
||||
<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
|
||||
<span class="n">gradients</span> <span class="o">=</span> <span class="n">training_gradient</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
|
||||
<span class="n">theta</span> <span class="o">-=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="nb">iter</span><span class="p">,</span><span class="n">gradients</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="n">gradients</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"theta from own gd"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Now improve with momentum gradient descent</span>
|
||||
<span class="n">change</span> <span class="o">=</span> <span class="mf">0.0</span>
|
||||
<span class="n">delta_momentum</span> <span class="o">=</span> <span class="mf">0.3</span>
|
||||
<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
|
||||
<span class="c1"># calculate gradient</span>
|
||||
<span class="n">gradients</span> <span class="o">=</span> <span class="n">training_gradient</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
|
||||
<span class="c1"># calculate update</span>
|
||||
<span class="n">new_change</span> <span class="o">=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span><span class="o">+</span><span class="n">delta_momentum</span><span class="o">*</span><span class="n">change</span>
|
||||
<span class="c1"># take a step</span>
|
||||
<span class="n">theta</span> <span class="o">-=</span> <span class="n">new_change</span>
|
||||
<span class="c1"># save the change</span>
|
||||
<span class="n">change</span> <span class="o">=</span> <span class="n">new_change</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="nb">iter</span><span class="p">,</span><span class="n">gradients</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="n">gradients</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"theta from own gd wth momentum"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>We note indeed a considerable increase in efficiency here, we less iterations needed.
|
||||
However, if we can invert the Hessian matrix, this is the preferred approach, as shown in the example here.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Newton's method</span>
|
||||
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">beta</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">beta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
|
||||
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
|
||||
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
|
||||
<span class="n">beta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Own inversion"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">beta_linreg</span><span class="p">)</span>
|
||||
<span class="c1"># Hessian matrix</span>
|
||||
<span class="n">H</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span> <span class="n">XT_X</span>
|
||||
<span class="c1"># Note that here the Hessian does not depend on the parameters beta</span>
|
||||
<span class="n">invH</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
|
||||
<span class="n">EigValues</span><span class="p">,</span> <span class="n">EigVectors</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">eig</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Eigenvalues of Hessian Matrix:</span><span class="si">{</span><span class="n">EigValues</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
|
||||
|
||||
<span class="n">beta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">Niterations</span> <span class="o">=</span> <span class="mi">5</span>
|
||||
|
||||
<span class="c1"># define the gradient</span>
|
||||
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">)</span>
|
||||
|
||||
<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
|
||||
<span class="n">gradients</span> <span class="o">=</span> <span class="n">training_gradient</span><span class="p">(</span><span class="n">beta</span><span class="p">)</span>
|
||||
<span class="n">beta</span> <span class="o">-=</span> <span class="n">invH</span> <span class="o">@</span> <span class="n">gradients</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="nb">iter</span><span class="p">,</span><span class="n">gradients</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="n">gradients</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"beta from own Newton code"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">beta</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="including-stochastic-gradient-descent-with-autograd">
|
||||
<h3><span class="section-number">7.11.1. </span>Including Stochastic Gradient Descent with Autograd<a class="headerlink" href="#including-stochastic-gradient-descent-with-autograd" title="Permalink to this headline">¶</a></h3>
|
||||
<h2><span class="section-number">7.14. </span>Including Stochastic Gradient Descent with Autograd<a class="headerlink" href="#including-stochastic-gradient-descent-with-autograd" title="Permalink to this headline">¶</a></h2>
|
||||
<p>In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using <strong>autograd</strong>.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -2444,48 +2601,230 @@ first example shows results with ordinary leats squares.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="and-logistic-regression">
|
||||
<h3><span class="section-number">7.11.2. </span>And Logistic Regression<a class="headerlink" href="#and-logistic-regression" title="Permalink to this headline">¶</a></h3>
|
||||
<p>Here we include momentum in the standard gradient descent approach.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients using SGD</span>
|
||||
<span class="c1"># OLS example</span>
|
||||
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">sigmoid</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="mf">0.5</span> <span class="o">*</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">tanh</span><span class="p">(</span><span class="n">x</span> <span class="o">/</span> <span class="mf">2.</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="c1"># Note change from previous example</span>
|
||||
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">theta</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">logistic_predictions</span><span class="p">(</span><span class="n">weights</span><span class="p">,</span> <span class="n">inputs</span><span class="p">):</span>
|
||||
<span class="c1"># Outputs probability of a label being true according to logistic model.</span>
|
||||
<span class="k">return</span> <span class="n">sigmoid</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">weights</span><span class="p">))</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">training_loss</span><span class="p">(</span><span class="n">weights</span><span class="p">):</span>
|
||||
<span class="c1"># Training loss is the negative log-likelihood of the training labels.</span>
|
||||
<span class="n">preds</span> <span class="o">=</span> <span class="n">logistic_predictions</span><span class="p">(</span><span class="n">weights</span><span class="p">,</span> <span class="n">inputs</span><span class="p">)</span>
|
||||
<span class="n">label_probabilities</span> <span class="o">=</span> <span class="n">preds</span> <span class="o">*</span> <span class="n">targets</span> <span class="o">+</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">preds</span><span class="p">)</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">targets</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="o">-</span><span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">label_probabilities</span><span class="p">))</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
|
||||
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
|
||||
<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Own inversion"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
|
||||
<span class="c1"># Hessian matrix</span>
|
||||
<span class="n">H</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span> <span class="n">XT_X</span>
|
||||
<span class="n">EigValues</span><span class="p">,</span> <span class="n">EigVectors</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">eig</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Eigenvalues of Hessian Matrix:</span><span class="si">{</span><span class="n">EigValues</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Build a toy dataset.</span>
|
||||
<span class="n">inputs</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mf">0.52</span><span class="p">,</span> <span class="mf">1.12</span><span class="p">,</span> <span class="mf">0.77</span><span class="p">],</span>
|
||||
<span class="p">[</span><span class="mf">0.88</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.08</span><span class="p">,</span> <span class="mf">0.15</span><span class="p">],</span>
|
||||
<span class="p">[</span><span class="mf">0.52</span><span class="p">,</span> <span class="mf">0.06</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.30</span><span class="p">],</span>
|
||||
<span class="p">[</span><span class="mf">0.74</span><span class="p">,</span> <span class="o">-</span><span class="mf">2.49</span><span class="p">,</span> <span class="mf">1.39</span><span class="p">]])</span>
|
||||
<span class="n">targets</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="kc">False</span><span class="p">,</span> <span class="kc">True</span><span class="p">])</span>
|
||||
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">eta</span> <span class="o">=</span> <span class="mf">1.0</span><span class="o">/</span><span class="n">np</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">EigValues</span><span class="p">)</span>
|
||||
<span class="n">Niterations</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
|
||||
<span class="c1"># Define a function that returns gradients of training loss using Autograd.</span>
|
||||
<span class="n">training_gradient_fun</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">training_loss</span><span class="p">)</span>
|
||||
<span class="c1"># Note that we request the derivative wrt third argument (theta, 2 here)</span>
|
||||
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Optimize weights using gradient descent.</span>
|
||||
<span class="n">weights</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">])</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Initial loss:"</span><span class="p">,</span> <span class="n">training_loss</span><span class="p">(</span><span class="n">weights</span><span class="p">))</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
|
||||
<span class="n">weights</span> <span class="o">-=</span> <span class="n">training_gradient_fun</span><span class="p">(</span><span class="n">weights</span><span class="p">)</span> <span class="o">*</span> <span class="mf">0.01</span>
|
||||
<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
|
||||
<span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
|
||||
<span class="n">theta</span> <span class="o">-=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"theta from own gd"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
|
||||
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Trained loss:"</span><span class="p">,</span> <span class="n">training_loss</span><span class="p">(</span><span class="n">weights</span><span class="p">))</span>
|
||||
|
||||
<span class="n">n_epochs</span> <span class="o">=</span> <span class="mi">50</span>
|
||||
<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span> <span class="c1">#size of each minibatch</span>
|
||||
<span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n</span><span class="o">/</span><span class="n">M</span><span class="p">)</span> <span class="c1">#number of minibatches</span>
|
||||
<span class="n">t0</span><span class="p">,</span> <span class="n">t1</span> <span class="o">=</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">50</span>
|
||||
<span class="k">def</span> <span class="nf">learning_schedule</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">t0</span><span class="o">/</span><span class="p">(</span><span class="n">t</span><span class="o">+</span><span class="n">t1</span><span class="p">)</span>
|
||||
|
||||
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
|
||||
<span class="n">change</span> <span class="o">=</span> <span class="mf">0.0</span>
|
||||
<span class="n">delta_momentum</span> <span class="o">=</span> <span class="mf">0.3</span>
|
||||
|
||||
<span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_epochs</span><span class="p">):</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
|
||||
<span class="n">random_index</span> <span class="o">=</span> <span class="n">M</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
|
||||
<span class="n">xi</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
|
||||
<span class="n">yi</span> <span class="o">=</span> <span class="n">y</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
|
||||
<span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">M</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">yi</span><span class="p">,</span> <span class="n">xi</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
|
||||
<span class="n">eta</span> <span class="o">=</span> <span class="n">learning_schedule</span><span class="p">(</span><span class="n">epoch</span><span class="o">*</span><span class="n">m</span><span class="o">+</span><span class="n">i</span><span class="p">)</span>
|
||||
<span class="c1"># calculate update</span>
|
||||
<span class="n">new_change</span> <span class="o">=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span><span class="o">+</span><span class="n">delta_momentum</span><span class="o">*</span><span class="n">change</span>
|
||||
<span class="c1"># take a step</span>
|
||||
<span class="n">theta</span> <span class="o">-=</span> <span class="n">new_change</span>
|
||||
<span class="c1"># save the change</span>
|
||||
<span class="n">change</span> <span class="o">=</span> <span class="n">new_change</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"theta from own sdg with momentum"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="similar-second-order-function-now-problem-but-now-with-adagrad">
|
||||
<h3><span class="section-number">7.14.1. </span>Similar (second order function now) problem but now with AdaGrad<a class="headerlink" href="#similar-second-order-function-now-problem-but-now-with-adagrad" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent</span>
|
||||
<span class="c1"># OLS example</span>
|
||||
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
|
||||
|
||||
<span class="c1"># Note change from previous example</span>
|
||||
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">theta</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
|
||||
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">10000</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span> <span class="o">+</span><span class="mi">4</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="c1"># +np.random.randn(n,1)</span>
|
||||
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">,</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="p">]</span>
|
||||
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
|
||||
<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Own inversion"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="c1"># Note that we request the derivative wrt third argument (theta, 2 here)</span>
|
||||
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="c1"># Define parameters for Stochastic Gradient Descent</span>
|
||||
<span class="n">n_epochs</span> <span class="o">=</span> <span class="mi">50</span>
|
||||
<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span> <span class="c1">#size of each minibatch</span>
|
||||
<span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n</span><span class="o">/</span><span class="n">M</span><span class="p">)</span> <span class="c1">#number of minibatches</span>
|
||||
<span class="c1"># Guess for unknown parameters theta</span>
|
||||
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Value for learning rate</span>
|
||||
<span class="n">eta</span> <span class="o">=</span> <span class="mf">0.01</span>
|
||||
<span class="c1"># Including AdaGrad parameter to avoid possible division by zero</span>
|
||||
<span class="n">delta</span> <span class="o">=</span> <span class="mf">1e-8</span>
|
||||
<span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_epochs</span><span class="p">):</span>
|
||||
<span class="c1"># The outer product is calculated from scratch for each epoch</span>
|
||||
<span class="n">Giter</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
|
||||
<span class="n">random_index</span> <span class="o">=</span> <span class="n">M</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
|
||||
<span class="n">xi</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
|
||||
<span class="n">yi</span> <span class="o">=</span> <span class="n">y</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
|
||||
<span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">M</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">yi</span><span class="p">,</span> <span class="n">xi</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
|
||||
<span class="c1"># Calculate the outer product of the gradients</span>
|
||||
<span class="n">Giter</span> <span class="o">+=</span><span class="n">gradients</span> <span class="o">@</span> <span class="n">gradients</span><span class="o">.</span><span class="n">T</span>
|
||||
<span class="c1"># Simpler algorithm with only diagonal elements</span>
|
||||
<span class="n">Ginverse</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">eta</span><span class="o">/</span><span class="p">(</span><span class="n">delta</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">diagonal</span><span class="p">(</span><span class="n">Giter</span><span class="p">)))]</span>
|
||||
<span class="c1"># compute update</span>
|
||||
<span class="n">update</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">multiply</span><span class="p">(</span><span class="n">Ginverse</span><span class="p">,</span><span class="n">gradients</span><span class="p">)</span>
|
||||
<span class="n">theta</span> <span class="o">-=</span> <span class="n">update</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"theta from own AdaGrad"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Running this code we note an almost perfect agreement with the results from matrix inversion.</p>
|
||||
<p>Similarly, here is our implementation of RMSprop.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent</span>
|
||||
<span class="c1"># OLS example</span>
|
||||
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
|
||||
|
||||
<span class="c1"># Note change from previous example</span>
|
||||
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">theta</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
|
||||
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">10000</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span> <span class="o">+</span><span class="mi">4</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="c1"># +np.random.randn(n,1)</span>
|
||||
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">,</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="p">]</span>
|
||||
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
|
||||
<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Own inversion"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="c1"># Note that we request the derivative wrt third argument (theta, 2 here)</span>
|
||||
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="c1"># Define parameters for Stochastic Gradient Descent</span>
|
||||
<span class="n">n_epochs</span> <span class="o">=</span> <span class="mi">50</span>
|
||||
<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span> <span class="c1">#size of each minibatch</span>
|
||||
<span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n</span><span class="o">/</span><span class="n">M</span><span class="p">)</span> <span class="c1">#number of minibatches</span>
|
||||
<span class="c1"># Guess for unknown parameters theta</span>
|
||||
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Value for learning rate</span>
|
||||
<span class="n">eta</span> <span class="o">=</span> <span class="mf">0.01</span>
|
||||
<span class="c1"># Value for parameter rho</span>
|
||||
<span class="n">rho</span> <span class="o">=</span> <span class="mf">0.99</span>
|
||||
<span class="c1"># Including AdaGrad parameter to avoid possible division by zero</span>
|
||||
<span class="n">delta</span> <span class="o">=</span> <span class="mf">1e-8</span>
|
||||
<span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_epochs</span><span class="p">):</span>
|
||||
<span class="n">Giter</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
|
||||
<span class="n">random_index</span> <span class="o">=</span> <span class="n">M</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
|
||||
<span class="n">xi</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
|
||||
<span class="n">yi</span> <span class="o">=</span> <span class="n">y</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
|
||||
<span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">M</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">yi</span><span class="p">,</span> <span class="n">xi</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
|
||||
<span class="c1"># Previous value for the outer product of gradients</span>
|
||||
<span class="n">Previous</span> <span class="o">=</span> <span class="n">Giter</span>
|
||||
<span class="c1"># Accumulated gradient</span>
|
||||
<span class="n">Giter</span> <span class="o">+=</span><span class="n">gradients</span> <span class="o">@</span> <span class="n">gradients</span><span class="o">.</span><span class="n">T</span>
|
||||
<span class="c1"># Scaling with rho the new and the previous results</span>
|
||||
<span class="n">Gnew</span> <span class="o">=</span> <span class="p">(</span><span class="n">rho</span><span class="o">*</span><span class="n">Previous</span><span class="o">+</span><span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="n">rho</span><span class="p">)</span><span class="o">*</span><span class="n">Giter</span><span class="p">)</span>
|
||||
<span class="c1"># Taking the diagonal only and inverting</span>
|
||||
<span class="n">Ginverse</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">eta</span><span class="o">/</span><span class="p">(</span><span class="n">delta</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">diagonal</span><span class="p">(</span><span class="n">Gnew</span><span class="p">)))]</span>
|
||||
<span class="c1"># Hadamard product</span>
|
||||
<span class="n">update</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">multiply</span><span class="p">(</span><span class="n">Ginverse</span><span class="p">,</span><span class="n">gradients</span><span class="p">)</span>
|
||||
<span class="n">theta</span> <span class="o">-=</span> <span class="n">update</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"theta from own RMSprop"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="introducing-jax">
|
||||
<h2><span class="section-number">7.15. </span>Introducing <a class="reference external" href="https://jax.readthedocs.io/en/latest/">JAX</a><a class="headerlink" href="#introducing-jax" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Presently, instead of using <strong>autograd</strong>, we recommend using <a class="reference external" href="https://jax.readthedocs.io/en/latest/">JAX</a></p>
|
||||
<p><strong>JAX</strong> is Autograd and <a class="reference external" href="https://www.tensorflow.org/xla">XLA (Accelerated Linear Algebra))</a>,
|
||||
brought together for high-performance numerical computing and machine learning research.
|
||||
It provides composable transformations of Python+NumPy programs: differentiate, vectorize, parallelize, Just-In-Time compile to GPU/TPU, and more.</p>
|
||||
<p>Here’s a simple example on how you can use <strong>JAX</strong> to compute the derivate of the logistic function.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">jax.numpy</span> <span class="k">as</span> <span class="nn">jnp</span>
|
||||
<span class="kn">from</span> <span class="nn">jax</span> <span class="kn">import</span> <span class="n">grad</span><span class="p">,</span> <span class="n">jit</span><span class="p">,</span> <span class="n">vmap</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">sum_logistic</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">jnp</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="mf">1.0</span> <span class="o">/</span> <span class="p">(</span><span class="mf">1.0</span> <span class="o">+</span> <span class="n">jnp</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="p">)))</span>
|
||||
|
||||
<span class="n">x_small</span> <span class="o">=</span> <span class="n">jnp</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mf">3.</span><span class="p">)</span>
|
||||
<span class="n">derivative_fn</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">sum_logistic</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">derivative_fn</span><span class="p">(</span><span class="n">x_small</span><span class="p">))</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -87,7 +87,7 @@
|
||||
# Machine learning is an extremely rich field, in spite of its young
|
||||
# age. The increases we have seen during the last three decades in
|
||||
# computational capabilities have been followed by developments of
|
||||
# methods and techniques for analyzing and handling large date sets,
|
||||
# methods and techniques for analyzing and handling large data sets,
|
||||
# relying heavily on statistics, computer science and mathematics. The
|
||||
# field is rather new and developing rapidly. Popular software packages
|
||||
# written in Python for machine learning like
|
||||
@@ -110,7 +110,7 @@
|
||||
# problem, and let the computer deduce the logic behind it. On the other
|
||||
# hand, *unsupervised learning* is a method for finding patterns and
|
||||
# relationship in data sets without any prior knowledge of the system.
|
||||
# Some authours also operate with a third category, namely
|
||||
# Some authors also operate with a third category, namely
|
||||
# *reinforcement learning*. This is a paradigm of learning inspired by
|
||||
# behavioral psychology, where learning is achieved by trial-and-error,
|
||||
# solely from rewards and punishment.
|
||||
@@ -166,14 +166,14 @@
|
||||
# In science and engineering we often end up in situations where we want to infer (or learn) a
|
||||
# quantitative model $M$ for a given set of sample points $\boldsymbol{X} \in [x_1, x_2,\dots x_N]$.
|
||||
#
|
||||
# As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a
|
||||
# As we will see repeatedly in these lectures, we could try to fit these data points to a model given by a
|
||||
# straight line, or if we wish to be more sophisticated to a more complex
|
||||
# function.
|
||||
#
|
||||
# The reason for inferring such a model is that it
|
||||
# serves many useful purposes. On the one hand, the model can reveal information
|
||||
# encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important
|
||||
# corelations that relate interesting physics interpretations.
|
||||
# correlations that relate interesting physics interpretations.
|
||||
#
|
||||
# In addition, it can simplify the representation of the given data set and help
|
||||
# us in making predictions about future data samples.
|
||||
@@ -304,7 +304,7 @@ plt.show()
|
||||
# where $x$ is defined as before. Does the fit look better? Indeed, by
|
||||
# reducing the role of the noise given by the normal distribution we see immediately that
|
||||
# our linear prediction seemingly reproduces better the training
|
||||
# set. However, this testing 'by the eye' is obviouly not satisfactory in the
|
||||
# set. However, this testing 'by the eye' is obviously not satisfactory in the
|
||||
# long run. Here we have only defined the training data and our model, and
|
||||
# have not discussed a more rigorous approach to the **cost** function.
|
||||
#
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 18 KiB After Width: | Height: | Size: 18 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 25 KiB After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 17 KiB |
File diff suppressed because it is too large
Load Diff
@@ -88,7 +88,7 @@
|
||||
# This is given by the **Singular Value Decomposition** (SVD) algorithm,
|
||||
# perhaps the most powerful linear algebra algorithm. The SVD provides
|
||||
# a numerically stable matrix decomposition that is used in a large
|
||||
# swath oc applications and the decomposition is always stable
|
||||
# swath of applications and the decomposition is always stable
|
||||
# numerically.
|
||||
#
|
||||
# In machine learning it plays a central role in dealing with for
|
||||
@@ -104,7 +104,7 @@
|
||||
# are problems with near singular or singular matrices. The column vectors of $\boldsymbol{X}$
|
||||
# may be linearly dependent, normally referred to as super-collinearity.
|
||||
# This means that the matrix may be rank deficient and it is basically impossible to
|
||||
# to model the data using linear regression. As an example, consider the matrix
|
||||
# model the data using linear regression. As an example, consider the matrix
|
||||
|
||||
# $$
|
||||
# \begin{align*}
|
||||
@@ -125,7 +125,7 @@
|
||||
# the first column is the row-wise sum of the other two columns. The rank (more correct,
|
||||
# the column rank) of a matrix is the dimension of the space spanned by the
|
||||
# column vectors. Hence, the rank of $\mathbf{X}$ is equal to the number
|
||||
# of linearly independent columns. In this particular case the matrix has rank 2.
|
||||
# of linearly independent columns. In this particular case the matrix has rank 1.
|
||||
#
|
||||
# Super-collinearity of an $(n \times p)$-dimensional design matrix $\mathbf{X}$ implies
|
||||
# that the inverse of the matrix $\boldsymbol{X}^T\boldsymbol{X}$ (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this
|
||||
@@ -158,7 +158,7 @@
|
||||
|
||||
# has linearly dependent column vectors, we will not be able to compute the inverse
|
||||
# of $\boldsymbol{X}^T\boldsymbol{X}$ and we cannot find the parameters (estimators) $\beta_i$.
|
||||
# The estimators are only well-defined if $(\boldsymbol{X}^{T}\boldsymbol{X})^{-1}$ exits.
|
||||
# The estimators are only well-defined if $(\boldsymbol{X}^{T}\boldsymbol{X})$ can be inverted.
|
||||
# This is more likely to happen when the matrix $\boldsymbol{X}$ is high-dimensional. In this case it is likely to encounter a situation where
|
||||
# the regression parameters $\beta_i$ cannot be estimated.
|
||||
#
|
||||
@@ -172,7 +172,7 @@
|
||||
|
||||
# ## Basic math of the SVD
|
||||
#
|
||||
# From standard linear algebra we know that a square matrix $\boldsymbol{X}$ can be diagonalized if and only it is
|
||||
# From standard linear algebra we know that a square matrix $\boldsymbol{X}$ can be diagonalized if and only if it is
|
||||
# a so-called [normal matrix](https://en.wikipedia.org/wiki/Normal_matrix), that is if $\boldsymbol{X}\in {\mathbb{R}}^{n\times n}$
|
||||
# we have $\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X}$ or if $\boldsymbol{X}\in {\mathbb{C}}^{n\times n}$ we have $\boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X}$.
|
||||
# The matrix has then a set of eigenpairs
|
||||
@@ -359,7 +359,6 @@ def SVDinv(A):
|
||||
return np.matmul(V,np.matmul(invD,UT))
|
||||
|
||||
|
||||
#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])
|
||||
# Non-singular square matrix
|
||||
X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])
|
||||
print(X)
|
||||
@@ -378,7 +377,7 @@ print(np.abs(B-C))
|
||||
# It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.
|
||||
# It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.
|
||||
#
|
||||
# Using the SVD we can obtain the pseudoinverse of a matrix $\boldsymbol{A}$ (labeled here as $\boldsymbol{A}_{\mathrm{PI}}$
|
||||
# Using the SVD we can obtain the pseudoinverse (PI) of a matrix $\boldsymbol{A}$ (labeled here as $\boldsymbol{A}_{\mathrm{PI}}$
|
||||
|
||||
# $$
|
||||
# \boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T,
|
||||
@@ -441,7 +440,7 @@ print(np.abs(C-B))
|
||||
|
||||
# where $\boldsymbol{U}$ is an orthogonal matrix of dimension $n\times n$, meaning that $\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n$. Here $\boldsymbol{I}_n$ is the unit matrix of dimension $n \times n$.
|
||||
#
|
||||
# Similarly, $\boldsymbol{V}$ is an orthogonal matrix of dimension $p\times p$, meaning that $\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p$. Here $\boldsymbol{I}_n$ is the unit matrix of dimension $p \times p$.
|
||||
# Similarly, $\boldsymbol{V}$ is an orthogonal matrix of dimension $p\times p$, meaning that $\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p$. Here $\boldsymbol{I}_p$ is the unit matrix of dimension $p \times p$.
|
||||
#
|
||||
# Finally $\boldsymbol{\Sigma}$ contains the singular values $\sigma_i$. This matrix has dimension $n\times p$ and the singular values $\sigma_i$ are all positive. The non-zero values are ordered in descending order, that is
|
||||
|
||||
@@ -603,7 +602,7 @@ print(np.abs(C-B))
|
||||
# \frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
# $$
|
||||
|
||||
# This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
|
||||
# This quantity defines what is called the Hessian matrix (the second derivative of the cost function we want to optimize).
|
||||
#
|
||||
# The Hessian matrix plays an important role and is defined in this course as
|
||||
|
||||
@@ -709,7 +708,7 @@ print(np.abs(C-B))
|
||||
# $$
|
||||
|
||||
# With these definitions, we can now rewrite our $2\times 2$
|
||||
# correlation/covariance matrix in terms of a moe general design/feature
|
||||
# correlation/covariance matrix in terms of a more general design/feature
|
||||
# matrix $\boldsymbol{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$
|
||||
# covariance matrix for the vectors $\boldsymbol{x}_i$ with $i=0,1,\dots,p-1$
|
||||
|
||||
@@ -810,7 +809,7 @@ print(C)
|
||||
#
|
||||
# The above procedure with **numpy** can be made more compact if we use **pandas**.
|
||||
#
|
||||
# We whow here how we can set up the correlation matrix using **pandas**, as done in this simple code
|
||||
# We know here how we can set up the correlation matrix using **pandas**, as done in this simple code
|
||||
|
||||
# In[7]:
|
||||
|
||||
@@ -1208,7 +1207,7 @@ print(covariance_matrix)
|
||||
# Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
|
||||
# diagonal. In this case we have an equal number of rows and columns $n=p$.
|
||||
#
|
||||
# Our model approximation is just $\tilde{\boldsymbol{y}}=\boldsymbol{\beta}$ and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term $1/n$)
|
||||
# Our model approximation is just $\tilde{\boldsymbol{y}}=\boldsymbol{\beta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$)
|
||||
|
||||
# $$
|
||||
# C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
|
||||
@@ -1254,7 +1253,7 @@ print(covariance_matrix)
|
||||
|
||||
# Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the other hand the values of $\beta_i$ as function of $\lambda$.
|
||||
#
|
||||
# As another examples,
|
||||
# As another example,
|
||||
# let us assume we have a data set with outputs/targets given by the vector
|
||||
|
||||
# $$
|
||||
@@ -1443,13 +1442,13 @@ plt.legend()
|
||||
plt.show()
|
||||
|
||||
|
||||
# We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\boldsymbol{\beta}$, we that they are getting smaller and smaller and our error stabilizes since the predicted values of $\tilde{\boldsymbol{y}}$ approach zero.
|
||||
# We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\boldsymbol{\beta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\tilde{\boldsymbol{y}}$ approach zero.
|
||||
#
|
||||
# This happens also for Lasso regression, as seen from the next code
|
||||
# output. The difference is that Lasso shrinks the values of $\beta$ to
|
||||
# zero at a much earlier stage and the results flatten out. We see that
|
||||
# Lasso gives also an excellent fit for small values of $\lambda$ and
|
||||
# shows rthe best performance of the three regression methods.
|
||||
# shows the best performance of the three regression methods.
|
||||
|
||||
# In[10]:
|
||||
|
||||
@@ -1800,7 +1799,7 @@ plt.show()
|
||||
|
||||
# which reads as finding the likelihood of an event $y_i$ with the input variables $\boldsymbol{X}$ given the parameters (to be determined) $\boldsymbol{\beta}$.
|
||||
#
|
||||
# Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event $\boldsymbol{y}$ as the product of the single events, that is we have
|
||||
# Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event $\boldsymbol{y}$ as the product of the single events, that is we have
|
||||
|
||||
# $$
|
||||
# p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
|
||||
@@ -1888,7 +1887,7 @@ plt.show()
|
||||
# The product rule (aka joint probability) is given by
|
||||
|
||||
# $$
|
||||
# p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
|
||||
# p(X \cap Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
|
||||
# $$
|
||||
|
||||
# where we read $p(X\vert Y)$ as the likelihood of obtaining $X$ given $Y$.
|
||||
@@ -2161,7 +2160,7 @@ for i in range(nlambdas):
|
||||
# noise. Here we recommend to use $\sigma^2=1$ as variance for the
|
||||
# added noise (which follows a normal distribution with mean value zero).
|
||||
# Comment your results. If you have a large noise term, do the parameters $\beta_j$ vary more as function
|
||||
# model complexity? And what about their variance?
|
||||
# of model complexity? And what about their variance?
|
||||
|
||||
# ## Linking Bayes' Theorem with Ridge and Lasso Regression
|
||||
#
|
||||
@@ -2195,7 +2194,7 @@ for i in range(nlambdas):
|
||||
# p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}).
|
||||
# $$
|
||||
|
||||
# We have a model for $p(\boldsymbol{D}\vert\boldsymbol{\beta})$ but need one for the **prior** $p(\boldsymbol{\beta}$!
|
||||
# We have a model for $p(\boldsymbol{D}\vert\boldsymbol{\beta})$ but need one for the **prior** $p(\boldsymbol{\beta})$!
|
||||
#
|
||||
# With the posterior probability defined by a likelihood which we have
|
||||
# already modeled and an unknown prior, we are now ready to make
|
||||
@@ -2249,13 +2248,13 @@ for i in range(nlambdas):
|
||||
# constants terms that do not depend on $\beta$, we have
|
||||
|
||||
# $$
|
||||
# C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
# C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
# $$
|
||||
|
||||
# and replacing $1/\tau$ with $\lambda$ we have
|
||||
|
||||
# $$
|
||||
# C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
# C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
# $$
|
||||
|
||||
# which is our Lasso cost function!
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -250,7 +250,19 @@
|
||||
# $\mathbb{R}$. Examples of convex sets of $\mathbb{R}^2$ are the
|
||||
# regular polygons (triangles, rectangles, pentagons, etc...).
|
||||
#
|
||||
# **Convex function**: Let $X \subset \mathbb{R}^n$ be a convex set. Assume that the function $f: X \rightarrow \mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \in X$ and for all $t \in [0,1]$. If $\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \neq x_2$ and $t\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below.
|
||||
# **Convex function**: Let $X \subset \mathbb{R}^n$ be a convex
|
||||
# set. Assume that the function $f: X \rightarrow \mathbb{R}$ is
|
||||
# continuous, then $f$ is said to be convex if
|
||||
# $f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2)$
|
||||
# for all
|
||||
# $x_1, x_2 \in X$ and for all $t \in [0,1]$.
|
||||
#
|
||||
# If $\leq$ is replaced with a strict inequality in the
|
||||
# definition, we demand $x_1 \neq x_2$ and $t\in(0,1)$ then $f$ is said
|
||||
# to be strictly convex. For a single variable function, convexity means
|
||||
# that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the
|
||||
# value of the function on the interval $[x_1,x_2]$ is always below the
|
||||
# line as discussed below.
|
||||
#
|
||||
# In the following we state first and second-order conditions which
|
||||
# ensures convexity of a function $f$. We write $D_f$ to denote the
|
||||
@@ -264,7 +276,7 @@
|
||||
# is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds
|
||||
# for all $x,y \in D_f$. This condition means that for a convex function
|
||||
# the first order Taylor expansion (right hand side above) at any point
|
||||
# a global under estimator of the function. To convince yourself you can
|
||||
# is a global under estimator of the function. To convince yourself you can
|
||||
# make a drawing of $f(x) = x^2+1$ and draw the tangent line to $f(x)$ and
|
||||
# note that it is always below the graph.
|
||||
#
|
||||
@@ -1742,7 +1754,14 @@ a*= b
|
||||
a /=b
|
||||
|
||||
|
||||
# ## Using Autograd with OLS
|
||||
# ## Replace or not
|
||||
#
|
||||
# In the above code, we have use replacement in setting up the
|
||||
# mini-batches. The discussion
|
||||
# [here](https://sebastianraschka.com/faq/docs/sgd-methods.html) may be
|
||||
# useful.
|
||||
|
||||
# ## Using Autograd
|
||||
#
|
||||
# We conclude the part on optmization by showing how we can make codes
|
||||
# for linear regression and logistic regression using **autograd**. The
|
||||
@@ -1802,13 +1821,118 @@ plt.title(r'Random numbers ')
|
||||
plt.show()
|
||||
|
||||
|
||||
# ### Including Stochastic Gradient Descent with Autograd
|
||||
#
|
||||
# In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**.
|
||||
# ## Same code but now with momentum gradient descent
|
||||
|
||||
# In[27]:
|
||||
|
||||
|
||||
# Using Autograd to calculate gradients for OLS
|
||||
from random import random, seed
|
||||
import numpy as np
|
||||
import autograd.numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
from autograd import grad
|
||||
|
||||
def CostOLS(beta):
|
||||
return (1.0/n)*np.sum((y-X @ beta)**2)
|
||||
|
||||
n = 100
|
||||
x = 2*np.random.rand(n,1)
|
||||
y = 4+3*x#+np.random.randn(n,1)
|
||||
|
||||
X = np.c_[np.ones((n,1)), x]
|
||||
XT_X = X.T @ X
|
||||
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
|
||||
print("Own inversion")
|
||||
print(theta_linreg)
|
||||
# Hessian matrix
|
||||
H = (2.0/n)* XT_X
|
||||
EigValues, EigVectors = np.linalg.eig(H)
|
||||
print(f"Eigenvalues of Hessian Matrix:{EigValues}")
|
||||
|
||||
theta = np.random.randn(2,1)
|
||||
eta = 1.0/np.max(EigValues)
|
||||
Niterations = 30
|
||||
|
||||
# define the gradient
|
||||
training_gradient = grad(CostOLS)
|
||||
|
||||
for iter in range(Niterations):
|
||||
gradients = training_gradient(theta)
|
||||
theta -= eta*gradients
|
||||
print(iter,gradients[0],gradients[1])
|
||||
print("theta from own gd")
|
||||
print(theta)
|
||||
|
||||
# Now improve with momentum gradient descent
|
||||
change = 0.0
|
||||
delta_momentum = 0.3
|
||||
for iter in range(Niterations):
|
||||
# calculate gradient
|
||||
gradients = training_gradient(theta)
|
||||
# calculate update
|
||||
new_change = eta*gradients+delta_momentum*change
|
||||
# take a step
|
||||
theta -= new_change
|
||||
# save the change
|
||||
change = new_change
|
||||
print(iter,gradients[0],gradients[1])
|
||||
print("theta from own gd wth momentum")
|
||||
print(theta)
|
||||
|
||||
|
||||
# We note indeed a considerable increase in efficiency here, we less iterations needed.
|
||||
# However, if we can invert the Hessian matrix, this is the preferred approach, as shown in the example here.
|
||||
|
||||
# In[28]:
|
||||
|
||||
|
||||
# Using Newton's method
|
||||
from random import random, seed
|
||||
import numpy as np
|
||||
import autograd.numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
from autograd import grad
|
||||
|
||||
def CostOLS(beta):
|
||||
return (1.0/n)*np.sum((y-X @ beta)**2)
|
||||
|
||||
n = 100
|
||||
x = 2*np.random.rand(n,1)
|
||||
y = 4+3*x+np.random.randn(n,1)
|
||||
|
||||
X = np.c_[np.ones((n,1)), x]
|
||||
XT_X = X.T @ X
|
||||
beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
|
||||
print("Own inversion")
|
||||
print(beta_linreg)
|
||||
# Hessian matrix
|
||||
H = (2.0/n)* XT_X
|
||||
# Note that here the Hessian does not depend on the parameters beta
|
||||
invH = np.linalg.pinv(H)
|
||||
EigValues, EigVectors = np.linalg.eig(H)
|
||||
print(f"Eigenvalues of Hessian Matrix:{EigValues}")
|
||||
|
||||
beta = np.random.randn(2,1)
|
||||
Niterations = 5
|
||||
|
||||
# define the gradient
|
||||
training_gradient = grad(CostOLS)
|
||||
|
||||
for iter in range(Niterations):
|
||||
gradients = training_gradient(beta)
|
||||
beta -= invH @ gradients
|
||||
print(iter,gradients[0],gradients[1])
|
||||
print("beta from own Newton code")
|
||||
print(beta)
|
||||
|
||||
|
||||
# ## Including Stochastic Gradient Descent with Autograd
|
||||
# In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**.
|
||||
|
||||
# In[29]:
|
||||
|
||||
|
||||
# Using Autograd to calculate gradients using SGD
|
||||
# OLS example
|
||||
from random import random, seed
|
||||
@@ -1884,42 +2008,227 @@ print("theta from own sdg")
|
||||
print(theta)
|
||||
|
||||
|
||||
# ### And Logistic Regression
|
||||
# Here we include momentum in the standard gradient descent approach.
|
||||
|
||||
# In[28]:
|
||||
# In[30]:
|
||||
|
||||
|
||||
# Using Autograd to calculate gradients using SGD
|
||||
# OLS example
|
||||
from random import random, seed
|
||||
import numpy as np
|
||||
import autograd.numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
from autograd import grad
|
||||
|
||||
def sigmoid(x):
|
||||
return 0.5 * (np.tanh(x / 2.) + 1)
|
||||
# Note change from previous example
|
||||
def CostOLS(y,X,theta):
|
||||
return np.sum((y-X @ theta)**2)
|
||||
|
||||
def logistic_predictions(weights, inputs):
|
||||
# Outputs probability of a label being true according to logistic model.
|
||||
return sigmoid(np.dot(inputs, weights))
|
||||
n = 100
|
||||
x = 2*np.random.rand(n,1)
|
||||
y = 4+3*x+np.random.randn(n,1)
|
||||
|
||||
def training_loss(weights):
|
||||
# Training loss is the negative log-likelihood of the training labels.
|
||||
preds = logistic_predictions(weights, inputs)
|
||||
label_probabilities = preds * targets + (1 - preds) * (1 - targets)
|
||||
return -np.sum(np.log(label_probabilities))
|
||||
X = np.c_[np.ones((n,1)), x]
|
||||
XT_X = X.T @ X
|
||||
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
|
||||
print("Own inversion")
|
||||
print(theta_linreg)
|
||||
# Hessian matrix
|
||||
H = (2.0/n)* XT_X
|
||||
EigValues, EigVectors = np.linalg.eig(H)
|
||||
print(f"Eigenvalues of Hessian Matrix:{EigValues}")
|
||||
|
||||
# Build a toy dataset.
|
||||
inputs = np.array([[0.52, 1.12, 0.77],
|
||||
[0.88, -1.08, 0.15],
|
||||
[0.52, 0.06, -1.30],
|
||||
[0.74, -2.49, 1.39]])
|
||||
targets = np.array([True, True, False, True])
|
||||
theta = np.random.randn(2,1)
|
||||
eta = 1.0/np.max(EigValues)
|
||||
Niterations = 100
|
||||
|
||||
# Define a function that returns gradients of training loss using Autograd.
|
||||
training_gradient_fun = grad(training_loss)
|
||||
# Note that we request the derivative wrt third argument (theta, 2 here)
|
||||
training_gradient = grad(CostOLS,2)
|
||||
|
||||
# Optimize weights using gradient descent.
|
||||
weights = np.array([0.0, 0.0, 0.0])
|
||||
print("Initial loss:", training_loss(weights))
|
||||
for i in range(100):
|
||||
weights -= training_gradient_fun(weights) * 0.01
|
||||
for iter in range(Niterations):
|
||||
gradients = (1.0/n)*training_gradient(y, X, theta)
|
||||
theta -= eta*gradients
|
||||
print("theta from own gd")
|
||||
print(theta)
|
||||
|
||||
print("Trained loss:", training_loss(weights))
|
||||
|
||||
n_epochs = 50
|
||||
M = 5 #size of each minibatch
|
||||
m = int(n/M) #number of minibatches
|
||||
t0, t1 = 5, 50
|
||||
def learning_schedule(t):
|
||||
return t0/(t+t1)
|
||||
|
||||
theta = np.random.randn(2,1)
|
||||
|
||||
change = 0.0
|
||||
delta_momentum = 0.3
|
||||
|
||||
for epoch in range(n_epochs):
|
||||
for i in range(m):
|
||||
random_index = M*np.random.randint(m)
|
||||
xi = X[random_index:random_index+M]
|
||||
yi = y[random_index:random_index+M]
|
||||
gradients = (1.0/M)*training_gradient(yi, xi, theta)
|
||||
eta = learning_schedule(epoch*m+i)
|
||||
# calculate update
|
||||
new_change = eta*gradients+delta_momentum*change
|
||||
# take a step
|
||||
theta -= new_change
|
||||
# save the change
|
||||
change = new_change
|
||||
print("theta from own sdg with momentum")
|
||||
print(theta)
|
||||
|
||||
|
||||
# ### Similar (second order function now) problem but now with AdaGrad
|
||||
|
||||
# In[31]:
|
||||
|
||||
|
||||
# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent
|
||||
# OLS example
|
||||
from random import random, seed
|
||||
import numpy as np
|
||||
import autograd.numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
from autograd import grad
|
||||
|
||||
# Note change from previous example
|
||||
def CostOLS(y,X,theta):
|
||||
return np.sum((y-X @ theta)**2)
|
||||
|
||||
n = 10000
|
||||
x = np.random.rand(n,1)
|
||||
y = 2.0+3*x +4*x*x# +np.random.randn(n,1)
|
||||
|
||||
X = np.c_[np.ones((n,1)), x, x*x]
|
||||
XT_X = X.T @ X
|
||||
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
|
||||
print("Own inversion")
|
||||
print(theta_linreg)
|
||||
|
||||
|
||||
# Note that we request the derivative wrt third argument (theta, 2 here)
|
||||
training_gradient = grad(CostOLS,2)
|
||||
# Define parameters for Stochastic Gradient Descent
|
||||
n_epochs = 50
|
||||
M = 5 #size of each minibatch
|
||||
m = int(n/M) #number of minibatches
|
||||
# Guess for unknown parameters theta
|
||||
theta = np.random.randn(3,1)
|
||||
|
||||
# Value for learning rate
|
||||
eta = 0.01
|
||||
# Including AdaGrad parameter to avoid possible division by zero
|
||||
delta = 1e-8
|
||||
for epoch in range(n_epochs):
|
||||
# The outer product is calculated from scratch for each epoch
|
||||
Giter = np.zeros(shape=(3,3))
|
||||
for i in range(m):
|
||||
random_index = M*np.random.randint(m)
|
||||
xi = X[random_index:random_index+M]
|
||||
yi = y[random_index:random_index+M]
|
||||
gradients = (1.0/M)*training_gradient(yi, xi, theta)
|
||||
# Calculate the outer product of the gradients
|
||||
Giter +=gradients @ gradients.T
|
||||
# Simpler algorithm with only diagonal elements
|
||||
Ginverse = np.c_[eta/(delta+np.sqrt(np.diagonal(Giter)))]
|
||||
# compute update
|
||||
update = np.multiply(Ginverse,gradients)
|
||||
theta -= update
|
||||
print("theta from own AdaGrad")
|
||||
print(theta)
|
||||
|
||||
|
||||
# Running this code we note an almost perfect agreement with the results from matrix inversion.
|
||||
#
|
||||
# Similarly, here is our implementation of RMSprop.
|
||||
|
||||
# In[32]:
|
||||
|
||||
|
||||
# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent
|
||||
# OLS example
|
||||
from random import random, seed
|
||||
import numpy as np
|
||||
import autograd.numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
from autograd import grad
|
||||
|
||||
# Note change from previous example
|
||||
def CostOLS(y,X,theta):
|
||||
return np.sum((y-X @ theta)**2)
|
||||
|
||||
n = 10000
|
||||
x = np.random.rand(n,1)
|
||||
y = 2.0+3*x +4*x*x# +np.random.randn(n,1)
|
||||
|
||||
X = np.c_[np.ones((n,1)), x, x*x]
|
||||
XT_X = X.T @ X
|
||||
theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
|
||||
print("Own inversion")
|
||||
print(theta_linreg)
|
||||
|
||||
|
||||
# Note that we request the derivative wrt third argument (theta, 2 here)
|
||||
training_gradient = grad(CostOLS,2)
|
||||
# Define parameters for Stochastic Gradient Descent
|
||||
n_epochs = 50
|
||||
M = 5 #size of each minibatch
|
||||
m = int(n/M) #number of minibatches
|
||||
# Guess for unknown parameters theta
|
||||
theta = np.random.randn(3,1)
|
||||
|
||||
# Value for learning rate
|
||||
eta = 0.01
|
||||
# Value for parameter rho
|
||||
rho = 0.99
|
||||
# Including AdaGrad parameter to avoid possible division by zero
|
||||
delta = 1e-8
|
||||
for epoch in range(n_epochs):
|
||||
Giter = np.zeros(shape=(3,3))
|
||||
for i in range(m):
|
||||
random_index = M*np.random.randint(m)
|
||||
xi = X[random_index:random_index+M]
|
||||
yi = y[random_index:random_index+M]
|
||||
gradients = (1.0/M)*training_gradient(yi, xi, theta)
|
||||
# Previous value for the outer product of gradients
|
||||
Previous = Giter
|
||||
# Accumulated gradient
|
||||
Giter +=gradients @ gradients.T
|
||||
# Scaling with rho the new and the previous results
|
||||
Gnew = (rho*Previous+(1-rho)*Giter)
|
||||
# Taking the diagonal only and inverting
|
||||
Ginverse = np.c_[eta/(delta+np.sqrt(np.diagonal(Gnew)))]
|
||||
# Hadamard product
|
||||
update = np.multiply(Ginverse,gradients)
|
||||
theta -= update
|
||||
print("theta from own RMSprop")
|
||||
print(theta)
|
||||
|
||||
|
||||
# ## Introducing [JAX](https://jax.readthedocs.io/en/latest/)
|
||||
#
|
||||
# Presently, instead of using **autograd**, we recommend using [JAX](https://jax.readthedocs.io/en/latest/)
|
||||
#
|
||||
# **JAX** is Autograd and [XLA (Accelerated Linear Algebra))](https://www.tensorflow.org/xla),
|
||||
# brought together for high-performance numerical computing and machine learning research.
|
||||
# It provides composable transformations of Python+NumPy programs: differentiate, vectorize, parallelize, Just-In-Time compile to GPU/TPU, and more.
|
||||
#
|
||||
# Here's a simple example on how you can use **JAX** to compute the derivate of the logistic function.
|
||||
|
||||
# In[33]:
|
||||
|
||||
|
||||
import jax.numpy as jnp
|
||||
from jax import grad, jit, vmap
|
||||
|
||||
def sum_logistic(x):
|
||||
return jnp.sum(1.0 / (1.0 + jnp.exp(-x)))
|
||||
|
||||
x_small = jnp.arange(3.)
|
||||
derivative_fn = grad(sum_logistic)
|
||||
print(derivative_fn(x_small))
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 24 KiB After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 21 KiB After Width: | Height: | Size: 21 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 22 KiB After Width: | Height: | Size: 21 KiB |
Reference in New Issue
Block a user