lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cleaning up

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-11-03 05:58:42 +01:00
parent af8d066825
commit 2d6b94571f
7 changed files with 466 additions and 307 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week39/html/week39-bs.html
+37 -32
View File
@@ -147,6 +147,10 @@ doconce format html week39.do.txt --html_style=bootstrap --pygments_html_style=d
2,
None,
'gradient-descent-and-ridge'),
('The Hessian matrix for Ridge Regression',
2,
None,
'the-hessian-matrix-for-ridge-regression'),
('Program example for gradient descent with Ridge Regression',
2,
None,
@@ -292,36 +296,37 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week39-bs042.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#and-a-corresponding-example-using-scikit-learn" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#challenge-yourself" style="font-size: 80%;">Challenge yourself</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs048.html#friday-october-1" style="font-size: 80%;">Friday October 1</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs049.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs052.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs053.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs054.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs055.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs056.html#program-for-stochastic-gradient" style="font-size: 80%;">Program for stochastic gradient</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs057.html#momentum-based-gd" style="font-size: 80%;">Momentum based GD</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs058.html#more-on-momentum-based-approaches" style="font-size: 80%;">More on momentum based approaches</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs059.html#momentum-parameter" style="font-size: 80%;">Momentum parameter</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs060.html#second-moment-of-the-gradient" style="font-size: 80%;">Second moment of the gradient</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs061.html#rms-prop" style="font-size: 80%;">RMS prop</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs062.html#adam-optimizer" style="font-size: 80%;">ADAM optimizer</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs063.html#practical-tips" style="font-size: 80%;">Practical tips</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs064.html#automatic-differentiation" style="font-size: 80%;">Automatic differentiation</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs065.html#using-autograd" style="font-size: 80%;">Using autograd</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs066.html#autograd-with-more-complicated-functions" style="font-size: 80%;">Autograd with more complicated functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs067.html#more-complicated-functions-using-the-elements-of-their-arguments-directly" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs068.html#functions-using-mathematical-functions-from-numpy" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs069.html#more-autograd" style="font-size: 80%;">More autograd</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs070.html#and-with-loops" style="font-size: 80%;">And with loops</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs071.html#using-recursion" style="font-size: 80%;">Using recursion</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs072.html#unsupported-functions" style="font-size: 80%;">Unsupported functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs073.html#the-syntax-a-dot-b-when-finding-the-dot-product" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs074.html#recommended-to-avoid" style="font-size: 80%;">Recommended to avoid</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#the-hessian-matrix-for-ridge-regression" style="font-size: 80%;">The Hessian matrix for Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs048.html#challenge-yourself" style="font-size: 80%;">Challenge yourself</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs049.html#friday-october-1" style="font-size: 80%;">Friday October 1</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs052.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs053.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs054.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs055.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs056.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs057.html#program-for-stochastic-gradient" style="font-size: 80%;">Program for stochastic gradient</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs058.html#momentum-based-gd" style="font-size: 80%;">Momentum based GD</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs059.html#more-on-momentum-based-approaches" style="font-size: 80%;">More on momentum based approaches</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs060.html#momentum-parameter" style="font-size: 80%;">Momentum parameter</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs061.html#second-moment-of-the-gradient" style="font-size: 80%;">Second moment of the gradient</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs062.html#rms-prop" style="font-size: 80%;">RMS prop</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs063.html#adam-optimizer" style="font-size: 80%;">ADAM optimizer</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs064.html#practical-tips" style="font-size: 80%;">Practical tips</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs065.html#automatic-differentiation" style="font-size: 80%;">Automatic differentiation</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs066.html#using-autograd" style="font-size: 80%;">Using autograd</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs067.html#autograd-with-more-complicated-functions" style="font-size: 80%;">Autograd with more complicated functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs068.html#more-complicated-functions-using-the-elements-of-their-arguments-directly" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs069.html#functions-using-mathematical-functions-from-numpy" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs070.html#more-autograd" style="font-size: 80%;">More autograd</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs071.html#and-with-loops" style="font-size: 80%;">And with loops</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs072.html#using-recursion" style="font-size: 80%;">Using recursion</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs073.html#unsupported-functions" style="font-size: 80%;">Unsupported functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs074.html#the-syntax-a-dot-b-when-finding-the-dot-product" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs075.html#recommended-to-avoid" style="font-size: 80%;">Recommended to avoid</a></li>
</ul>
</li>
@@ -351,7 +356,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Nov 2, 2021</h4>
<h4>Nov 3, 2021</h4>
</center> <!-- date -->
<br>
@@ -376,7 +381,7 @@ MathJax.Hub.Config({
<li><a href="._week39-bs008.html">9</a></li>
<li><a href="._week39-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week39-bs074.html">75</a></li>
<li><a href="._week39-bs075.html">76</a></li>
<li><a href="._week39-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week39/html/week39-reveal.html
+30 -3
View File
@@ -184,7 +184,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Nov 2, 2021</h4>
<h4>Nov 3, 2021</h4>
</center> <!-- date -->
<br>
@@ -1760,6 +1760,26 @@ $$
<p>&nbsp;<br>
</section>
<section>
<h2 id="the-hessian-matrix-for-ridge-regression">The Hessian matrix for Ridge Regression </h2>
<p>The Hessian matrix of Ridge Regression for our simple example is given by </p>
<p>&nbsp;<br>
$$
\boldsymbol{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\
\end{bmatrix} = \frac{2}{n}X^T X+2\lambda\boldsymbol{I}.
$$
<p>&nbsp;<br>
<p>This implies that the Hessian matrix is positive definite, hence the stationary point is a
minimum.
Note that the Ridge loss function is convex, as a sum of two convex
functions. Therefore, the stationary point is a global
minimum of this function.
</p>
</section>
<section>
<h2 id="program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression </h2>
@@ -1787,14 +1807,21 @@ XT_X = X.T @ X
<span style="color: #228B22">#Ridge parameter lambda</span>
lmbda = <span style="color: #B452CD">0.001</span>
Id = lmbda* np.eye(XT_X.shape[<span style="color: #B452CD">0</span>])
Id = n*lmbda* np.eye(XT_X.shape[<span style="color: #B452CD">0</span>])
<span style="color: #228B22"># Hessian matrix</span>
H = (<span style="color: #B452CD">2.0</span>/n)* XT_X+<span style="color: #B452CD">2</span>*lmbda
<span style="color: #228B22"># Get the eigenvalues</span>
EigValues, EigVectors = np.linalg.eig(H)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;Eigenvalues of Hessian Matrix:{</span>EigValues<span style="color: #CD5555">}&quot;</span>)
beta_linreg = np.linalg.inv(XT_X+Id) @ X.T @ y
<span style="color: #658b00">print</span>(beta_linreg)
<span style="color: #228B22"># Start plain gradient descent</span>
beta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
eta = <span style="color: #B452CD">0.1</span>
eta = <span style="color: #B452CD">1.0</span>/np.max(EigValues)
Niterations = <span style="color: #B452CD">100</span>
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
doc/pub/week39/html/week39-solarized.html
+31 -3
View File
@@ -174,6 +174,10 @@ div.toc p,a {
2,
None,
'gradient-descent-and-ridge'),
('The Hessian matrix for Ridge Regression',
2,
None,
'the-hessian-matrix-for-ridge-regression'),
('Program example for gradient descent with Ridge Regression',
2,
None,
@@ -278,7 +282,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Nov 2, 2021</h4>
<h4>Nov 3, 2021</h4>
</center> <!-- date -->
<br>
@@ -1701,6 +1705,23 @@ $$
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-hessian-matrix-for-ridge-regression">The Hessian matrix for Ridge Regression </h2>
<p>The Hessian matrix of Ridge Regression for our simple example is given by </p>
$$
\boldsymbol{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\
\end{bmatrix} = \frac{2}{n}X^T X+2\lambda\boldsymbol{I}.
$$
<p>This implies that the Hessian matrix is positive definite, hence the stationary point is a
minimum.
Note that the Ridge loss function is convex, as a sum of two convex
functions. Therefore, the stationary point is a global
minimum of this function.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression </h2>
@@ -1728,14 +1749,21 @@ XT_X = X.T @ X
<span style="color: #228B22">#Ridge parameter lambda</span>
lmbda = <span style="color: #B452CD">0.001</span>
Id = lmbda* np.eye(XT_X.shape[<span style="color: #B452CD">0</span>])
Id = n*lmbda* np.eye(XT_X.shape[<span style="color: #B452CD">0</span>])
<span style="color: #228B22"># Hessian matrix</span>
H = (<span style="color: #B452CD">2.0</span>/n)* XT_X+<span style="color: #B452CD">2</span>*lmbda
<span style="color: #228B22"># Get the eigenvalues</span>
EigValues, EigVectors = np.linalg.eig(H)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;Eigenvalues of Hessian Matrix:{</span>EigValues<span style="color: #CD5555">}&quot;</span>)
beta_linreg = np.linalg.inv(XT_X+Id) @ X.T @ y
<span style="color: #658b00">print</span>(beta_linreg)
<span style="color: #228B22"># Start plain gradient descent</span>
beta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
eta = <span style="color: #B452CD">0.1</span>
eta = <span style="color: #B452CD">1.0</span>/np.max(EigValues)
Niterations = <span style="color: #B452CD">100</span>
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
doc/pub/week39/html/week39.html
+31 -3
View File
@@ -251,6 +251,10 @@ div.toc p,a {
2,
None,
'gradient-descent-and-ridge'),
('The Hessian matrix for Ridge Regression',
2,
None,
'the-hessian-matrix-for-ridge-regression'),
('Program example for gradient descent with Ridge Regression',
2,
None,
@@ -355,7 +359,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Nov 2, 2021</h4>
<h4>Nov 3, 2021</h4>
</center> <!-- date -->
<br>
@@ -1778,6 +1782,23 @@ $$
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-hessian-matrix-for-ridge-regression">The Hessian matrix for Ridge Regression </h2>
<p>The Hessian matrix of Ridge Regression for our simple example is given by </p>
$$
\boldsymbol{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\
\end{bmatrix} = \frac{2}{n}X^T X+2\lambda\boldsymbol{I}.
$$
<p>This implies that the Hessian matrix is positive definite, hence the stationary point is a
minimum.
Note that the Ridge loss function is convex, as a sum of two convex
functions. Therefore, the stationary point is a global
minimum of this function.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression </h2>
@@ -1805,14 +1826,21 @@ XT_X <span style="color: #666666">=</span> X<span style="color: #666666">.</span
<span style="color: #408080; font-style: italic">#Ridge parameter lambda</span>
lmbda <span style="color: #666666">=</span> <span style="color: #666666">0.001</span>
Id <span style="color: #666666">=</span> lmbda<span style="color: #666666">*</span> np<span style="color: #666666">.</span>eye(XT_X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>])
Id <span style="color: #666666">=</span> n<span style="color: #666666">*</span>lmbda<span style="color: #666666">*</span> np<span style="color: #666666">.</span>eye(XT_X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>])
<span style="color: #408080; font-style: italic"># Hessian matrix</span>
H <span style="color: #666666">=</span> (<span style="color: #666666">2.0/</span>n)<span style="color: #666666">*</span> XT_X<span style="color: #666666">+2*</span>lmbda
<span style="color: #408080; font-style: italic"># Get the eigenvalues</span>
EigValues, EigVectors <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>eig(H)
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;Eigenvalues of Hessian Matrix:</span><span style="color: #BB6688; font-weight: bold">{</span>EigValues<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">&quot;</span>)
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(XT_X<span style="color: #666666">+</span>Id) <span style="color: #666666">@</span> X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y
<span style="color: #008000">print</span>(beta_linreg)
<span style="color: #408080; font-style: italic"># Start plain gradient descent</span>
beta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)
eta <span style="color: #666666">=</span> <span style="color: #666666">0.1</span>
eta <span style="color: #666666">=</span> <span style="color: #666666">1.0/</span>np<span style="color: #666666">.</span>max(EigValues)
Niterations <span style="color: #666666">=</span> <span style="color: #666666">100</span>
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(Niterations):
doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week39/ipynb/week39.ipynb
+311 -264
View File
File diff suppressed because it is too large Load Diff
doc/src/week39/week39.do.txt
+26 -2
View File
@@ -1164,6 +1164,23 @@ We can easily extend our program to minimize $C_{\text{ridge}}(\beta)$ using gra
\]
!et
!split
===== The Hessian matrix for Ridge Regression =====
The Hessian matrix of Ridge Regression for our simple example is given by
!bt
\[
\bm{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\
\end{bmatrix} = \frac{2}{n}X^T X+2\lambda\bm{I}.
\]
!et
This implies that the Hessian matrix is positive definite, hence the stationary point is a
minimum.
Note that the Ridge loss function is convex, as a sum of two convex
functions. Therefore, the stationary point is a global
minimum of this function.
!split
===== Program example for gradient descent with Ridge Regression =====
@@ -1186,14 +1203,21 @@ XT_X = X.T @ X
#Ridge parameter lambda
lmbda = 0.001
Id = lmbda* np.eye(XT_X.shape[0])
Id = n*lmbda* np.eye(XT_X.shape[0])
# Hessian matrix
H = (2.0/n)* XT_X+2*lmbda
# Get the eigenvalues
EigValues, EigVectors = np.linalg.eig(H)
print(f"Eigenvalues of Hessian Matrix:{EigValues}")
beta_linreg = np.linalg.inv(XT_X+Id) @ X.T @ y
print(beta_linreg)
# Start plain gradient descent
beta = np.random.randn(2,1)
eta = 0.1
eta = 1.0/np.max(EigValues)
Niterations = 100
for iter in range(Niterations):