lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on week39

Browse Source
This commit is contained in:
mhjensen
2020-09-22 11:31:13 +02:00
parent a45b0a5015
commit 15b93178dd
26 changed files with 231 additions and 204 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week39/html/._week39-bs000.html
+1 -1
View File
@@ -301,7 +301,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Sep 22, 2020</h4></center> <!-- date -->
<br>
<p>
doc/pub/week39/html/._week39-bs001.html
+3
View File
@@ -289,6 +289,9 @@ MathJax.Hub.Config({
<li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
</ul>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_self">Aurelien Geron's chapter 4</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf" target="_self">Murphy sections 8.3 and 8.5</a>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
doc/pub/week39/html/._week39-bs002.html
+3
View File
@@ -284,6 +284,9 @@ MathJax.Hub.Config({
<h2 id="___sec1" class="anchor">Thursday September 24 </h2>
<p>
"Overview Video, why do we care about gradient methods?":""
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
doc/pub/week39/html/._week39-bs025.html
+2 -2
View File
@@ -305,7 +305,7 @@ y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((m,<span style="color: #666666">1</span>)), x]
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb))<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>T)<span style="color: #666666">.</span>dot(y)
<span style="color: #008000; font-weight: bold">print</span>(beta_linreg)
<span style="color: #008000">print</span>(beta_linreg)
beta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)
eta <span style="color: #666666">=</span> <span style="color: #666666">0.1</span>
@@ -315,7 +315,7 @@ Niterations <span style="color: #666666">=</span> <span style="color: #666666">1
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>dot(beta)<span style="color: #666666">-</span>y)
beta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradients
<span style="color: #008000; font-weight: bold">print</span>(beta)
<span style="color: #008000">print</span>(beta)
xnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">0</span>],[<span style="color: #666666">2</span>]])
xbnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)), xnew]
ypredict <span style="color: #666666">=</span> xbnew<span style="color: #666666">.</span>dot(beta)
doc/pub/week39/html/._week39-bs026.html
+3 -3
View File
@@ -298,10 +298,10 @@ y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)), x]
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb))<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>T)<span style="color: #666666">.</span>dot(y)
<span style="color: #008000; font-weight: bold">print</span>(beta_linreg)
sgdreg <span style="color: #666666">=</span> SGDRegressor(n_iter <span style="color: #666666">=</span> <span style="color: #666666">50</span>, penalty<span style="color: #666666">=</span><span style="color: #008000">None</span>, eta0<span style="color: #666666">=0.1</span>)
<span style="color: #008000">print</span>(beta_linreg)
sgdreg <span style="color: #666666">=</span> SGDRegressor(n_iter <span style="color: #666666">=</span> <span style="color: #666666">50</span>, penalty<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>, eta0<span style="color: #666666">=0.1</span>)
sgdreg<span style="color: #666666">.</span>fit(x,y<span style="color: #666666">.</span>ravel())
<span style="color: #008000; font-weight: bold">print</span>(sgdreg<span style="color: #666666">.</span>intercept_, sgdreg<span style="color: #666666">.</span>coef_)
<span style="color: #008000">print</span>(sgdreg<span style="color: #666666">.</span>intercept_, sgdreg<span style="color: #666666">.</span>coef_)
</pre></div>
<p>
<p>
doc/pub/week39/html/._week39-bs028.html
+7 -7
View File
@@ -300,14 +300,14 @@ x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(m,<span style="color: #666666">1</span>)
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((m,<span style="color: #666666">1</span>)), x]
XT_X <span style="color: #666666">=</span> xb<span style="color: #666666">.</span>T @ xb
XT_X <span style="color: #666666">=</span> xb<span style="color: #666666">.</span>T <span style="color: #666666">@</span> xb
<span style="color: #408080; font-style: italic">#Ridge parameter lambda</span>
lmbda <span style="color: #666666">=</span> <span style="color: #666666">0.001</span>
Id <span style="color: #666666">=</span> lmbda<span style="color: #666666">*</span> np<span style="color: #666666">.</span>eye(XT_X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>])
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(XT_X<span style="color: #666666">+</span>Id) @ xb<span style="color: #666666">.</span>T @ y
<span style="color: #008000; font-weight: bold">print</span>(beta_linreg)
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(XT_X<span style="color: #666666">+</span>Id) <span style="color: #666666">@</span> xb<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y
<span style="color: #008000">print</span>(beta_linreg)
<span style="color: #408080; font-style: italic"># Start plain gradient descent</span>
beta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)
@@ -315,12 +315,12 @@ eta <span style="color: #666666">=</span> <span style="color: #666666">0.1</span
Niterations <span style="color: #666666">=</span> <span style="color: #666666">100</span>
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(Niterations):
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T @ (xb @ (beta)<span style="color: #666666">-</span>y)<span style="color: #666666">+2*</span>lmbda<span style="color: #666666">*</span>beta
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T <span style="color: #666666">@</span> (xb <span style="color: #666666">@</span> (beta)<span style="color: #666666">-</span>y)<span style="color: #666666">+2*</span>lmbda<span style="color: #666666">*</span>beta
beta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradients
<span style="color: #008000; font-weight: bold">print</span>(beta)
ypredict <span style="color: #666666">=</span> xb @ beta
ypredict2 <span style="color: #666666">=</span> xb @ beta_linreg
<span style="color: #008000">print</span>(beta)
ypredict <span style="color: #666666">=</span> xb <span style="color: #666666">@</span> beta
ypredict2 <span style="color: #666666">=</span> xb <span style="color: #666666">@</span> beta_linreg
plt<span style="color: #666666">.</span>plot(x, ypredict, <span style="color: #BA2121">&quot;r-&quot;</span>)
plt<span style="color: #666666">.</span>plot(x, ypredict2, <span style="color: #BA2121">&quot;b-&quot;</span>)
plt<span style="color: #666666">.</span>plot(x, y ,<span style="color: #BA2121">&#39;ro&#39;</span>)
doc/pub/week39/html/._week39-bs037.html
+1 -1
View File
@@ -325,7 +325,7 @@ j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
gamma_j <span style="color: #666666">=</span> step_length(t,t0,t1)
j <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;gamma_j after </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> epochs: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> (n_epochs,gamma_j))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;gamma_j after </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> epochs: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> (n_epochs,gamma_j))
</pre></div>
<p>
<p>
doc/pub/week39/html/._week39-bs038.html
+11 -11
View File
@@ -300,12 +300,12 @@ y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((m,<span style="color: #666666">1</span>)), x]
theta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb))<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>T)<span style="color: #666666">.</span>dot(y)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Own inversion&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(theta_linreg)
sgdreg <span style="color: #666666">=</span> SGDRegressor(max_iter <span style="color: #666666">=</span> <span style="color: #666666">50</span>, penalty<span style="color: #666666">=</span><span style="color: #008000">None</span>, eta0<span style="color: #666666">=0.1</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Own inversion&quot;</span>)
<span style="color: #008000">print</span>(theta_linreg)
sgdreg <span style="color: #666666">=</span> SGDRegressor(max_iter <span style="color: #666666">=</span> <span style="color: #666666">50</span>, penalty<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>, eta0<span style="color: #666666">=0.1</span>)
sgdreg<span style="color: #666666">.</span>fit(x,y<span style="color: #666666">.</span>ravel())
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;sgdreg from scikit&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(sgdreg<span style="color: #666666">.</span>intercept_, sgdreg<span style="color: #666666">.</span>coef_)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;sgdreg from scikit&quot;</span>)
<span style="color: #008000">print</span>(sgdreg<span style="color: #666666">.</span>intercept_, sgdreg<span style="color: #666666">.</span>coef_)
theta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)
@@ -314,10 +314,10 @@ Niterations <span style="color: #666666">=</span> <span style="color: #666666">1
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(Niterations):
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T @ ((xb @ theta)<span style="color: #666666">-</span>y)
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T <span style="color: #666666">@</span> ((xb <span style="color: #666666">@</span> theta)<span style="color: #666666">-</span>y)
theta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradients
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;theta frm own gd&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(theta)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;theta frm own gd&quot;</span>)
<span style="color: #008000">print</span>(theta)
xnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">0</span>],[<span style="color: #666666">2</span>]])
xbnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)), xnew]
@@ -337,11 +337,11 @@ theta <span style="color: #666666">=</span> np<span style="color: #666666">.</sp
random_index <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randint(m)
xi <span style="color: #666666">=</span> xb[random_index:random_index<span style="color: #666666">+1</span>]
yi <span style="color: #666666">=</span> y[random_index:random_index<span style="color: #666666">+1</span>]
gradients <span style="color: #666666">=</span> <span style="color: #666666">2</span> <span style="color: #666666">*</span> xi<span style="color: #666666">.</span>T @ ((xi @ theta)<span style="color: #666666">-</span>yi)
gradients <span style="color: #666666">=</span> <span style="color: #666666">2</span> <span style="color: #666666">*</span> xi<span style="color: #666666">.</span>T <span style="color: #666666">@</span> ((xi <span style="color: #666666">@</span> theta)<span style="color: #666666">-</span>yi)
eta <span style="color: #666666">=</span> learning_schedule(epoch<span style="color: #666666">*</span>m<span style="color: #666666">+</span>i)
theta <span style="color: #666666">=</span> theta <span style="color: #666666">-</span> eta<span style="color: #666666">*</span>gradients
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;theta from own sdg&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(theta)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;theta from own sdg&quot;</span>)
<span style="color: #008000">print</span>(theta)
plt<span style="color: #666666">.</span>plot(xnew, ypredict, <span style="color: #BA2121">&quot;r-&quot;</span>)
plt<span style="color: #666666">.</span>plot(xnew, ypredict2, <span style="color: #BA2121">&quot;b-&quot;</span>)
doc/pub/week39/html/._week39-bs046.html
+1 -1
View File
@@ -362,7 +362,7 @@ plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The max absolute difference is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(np<span style="color: #666666">.</span>max(np<span style="color: #666666">.</span>abs(computed <span style="color: #666666">-</span> analytic))))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The max absolute difference is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(np<span style="color: #666666">.</span>max(np<span style="color: #666666">.</span>abs(computed <span style="color: #666666">-</span> analytic))))
</pre></div>
<p>
<p>
doc/pub/week39/html/._week39-bs047.html
+2 -2
View File
@@ -306,11 +306,11 @@ f1_grad <span style="color: #666666">=</span> grad(f1)
a <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
<span style="color: #408080; font-style: italic"># See the evaluated gradient at a using autograd:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> using autograd is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,f1_grad(a)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> using autograd is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,f1_grad(a)))
<span style="color: #408080; font-style: italic"># Compare with the analytical derivative, that is f1&#39;(x) = 3*x**2 </span>
grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">3*</span>a<span style="color: #666666">**2</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> by finding the analytic expression is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,grad_analytical))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> by finding the analytic expression is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,grad_analytical))
</pre></div>
<p>
<p>
doc/pub/week39/html/._week39-bs048.html
+7 -7
View File
@@ -306,8 +306,8 @@ f2_grad_x2 <span style="color: #666666">=</span> grad(f2,<span style="color: #66
x1 <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
x2 <span style="color: #666666">=</span> <span style="color: #666666">3.0</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Evaluating at x1 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">, x2 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x1,x2))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;-&quot;</span><span style="color: #666666">*30</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Evaluating at x1 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">, x2 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x1,x2))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;-&quot;</span><span style="color: #666666">*30</span>)
<span style="color: #408080; font-style: italic"># Compare with the analytical derivatives:</span>
@@ -318,13 +318,13 @@ f2_grad_x1_analytical <span style="color: #666666">=</span> <span style="color:
f2_grad_x2_analytical <span style="color: #666666">=</span> x1 <span style="color: #666666">-</span> <span style="color: #666666">5</span>
<span style="color: #408080; font-style: italic"># See the evaluated derivations:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000; font-weight: bold">print</span>()
<span style="color: #008000">print</span>()
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
</pre></div>
<p>
Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
doc/pub/week39/html/._week39-bs049.html
+2 -2
View File
@@ -297,13 +297,13 @@ f3_grad <span style="color: #666666">=</span> grad(f3)
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">0</span>,<span style="color: #666666">4</span>,<span style="color: #666666">5</span>)
<span style="color: #408080; font-style: italic"># Print the computed gradient:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed gradient of f3 is: &quot;</span>, f3_grad(x))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed gradient of f3 is: &quot;</span>, f3_grad(x))
<span style="color: #408080; font-style: italic"># The analytical gradient is: (2, 3, 5, 7, 22*x[4])</span>
f3_grad_analytical <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">2</span>, <span style="color: #666666">3</span>, <span style="color: #666666">5</span>, <span style="color: #666666">7</span>, <span style="color: #666666">22*</span>x[<span style="color: #666666">4</span>]])
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f3 is: &quot;</span>, f3_grad_analytical)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f3 is: &quot;</span>, f3_grad_analytical)
</pre></div>
<p>
Note that in this case, when sending an array as input argument, the
doc/pub/week39/html/._week39-bs050.html
+2 -2
View File
@@ -297,13 +297,13 @@ f4_grad <span style="color: #666666">=</span> grad(f4)
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad(x)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad(x)))
<span style="color: #408080; font-style: italic"># The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi</span>
f4_grad_analytical <span style="color: #666666">=</span> x<span style="color: #666666">/</span>np<span style="color: #666666">.</span>sqrt(<span style="color: #666666">1</span> <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(x) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>cos(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x)<span style="color: #666666">*2*</span>np<span style="color: #666666">.</span>pi
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad_analytical))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad_analytical))
</pre></div>
<p>
<p>
doc/pub/week39/html/._week39-bs051.html
+1 -1
View File
@@ -300,7 +300,7 @@ f5_grad <span style="color: #666666">=</span> grad(f5)
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f5 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f5_grad(x)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f5 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f5_grad(x)))
</pre></div>
<p>
<p>
doc/pub/week39/html/._week39-bs052.html
+3 -3
View File
@@ -309,8 +309,8 @@ f6_while_grad <span style="color: #666666">=</span> grad(f6_while)
x <span style="color: #666666">=</span> <span style="color: #666666">0.5</span>
<span style="color: #408080; font-style: italic"># Print the computed derivaties of f6_for and f6_while</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_for at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_for_grad(x)))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_while at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_while_grad(x)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_for at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_for_grad(x)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_while at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_while_grad(x)))
</pre></div>
<p>
@@ -323,7 +323,7 @@ f6_grad_analytical <span style="color: #666666">=</span> <span style="color: #66
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10</span>):
f6_grad_analytical <span style="color: #666666">+=</span> i<span style="color: #666666">*</span>x<span style="color: #666666">**</span>(i<span style="color: #666666">-1</span>)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f6 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_grad_analytical))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f6 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_grad_analytical))
</pre></div>
<p>
<p>
doc/pub/week39/html/._week39-bs053.html
+2 -2
View File
@@ -299,7 +299,7 @@ f7_grad <span style="color: #666666">=</span> grad(f7)
n <span style="color: #666666">=</span> <span style="color: #666666">2.0</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad(n)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad(n)))
<span style="color: #408080; font-style: italic"># The function f7 is an implementation of the factorial of n.</span>
<span style="color: #408080; font-style: italic"># By using the product rule, one can find that the derivative is:</span>
@@ -312,7 +312,7 @@ f7_grad_analytical <span style="color: #666666">=</span> <span style="color: #66
tmp <span style="color: #666666">*=</span> (n <span style="color: #666666">-</span> k)
f7_grad_analytical <span style="color: #666666">+=</span> tmp
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad_analytical))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad_analytical))
</pre></div>
<p>
Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
doc/pub/week39/html/._week39-bs054.html
+1 -1
View File
@@ -300,7 +300,7 @@ f8_grad <span style="color: #666666">=</span> grad(f8)
x <span style="color: #666666">=</span> <span style="color: #666666">8.4</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f8 is:&quot;</span>,f8_grad(x))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The derivative of f8 is:&quot;</span>,f8_grad(x))
</pre></div>
<p>
Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
doc/pub/week39/html/._week39-bs055.html
+2 -2
View File
@@ -296,7 +296,7 @@ f9_grad <span style="color: #666666">=</span> grad(f9)
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">0.0</span>])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f9 is:&quot;</span>,f9_grad(x))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The derivative of f9 is:&quot;</span>,f9_grad(x))
</pre></div>
<p>
Here we are told that the 'dot' function does not belong to Autograd's
@@ -316,7 +316,7 @@ f9_alternative_grad <span style="color: #666666">=</span> grad(f9_alternative)
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">3.0</span>,<span style="color: #666666">0.0</span>])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The gradient of f9 is:&quot;</span>,f9_alternative_grad(x))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The gradient of f9 is:&quot;</span>,f9_alternative_grad(x))
<span style="color: #408080; font-style: italic"># The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively</span>
<span style="color: #408080; font-style: italic"># w.r.t x is (b_1, b_2).</span>
doc/pub/week39/html/._week39-bs065.html
+2 -2
View File
@@ -304,7 +304,7 @@ MathJax.Hub.Config({
fig <span style="color: #666666">=</span> pt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>gca(projection<span style="color: #666666">=</span><span style="color: #BA2121">&quot;3d&quot;</span>)
xmesh, ymesh <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mgrid[<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>,<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>]
xmesh, ymesh <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mgrid[<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50</span>j,<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50</span>j]
fmesh <span style="color: #666666">=</span> f(np<span style="color: #666666">.</span>array([xmesh, ymesh]))
ax<span style="color: #666666">.</span>plot_surface(xmesh, ymesh, fmesh)
</pre></div>
@@ -336,7 +336,7 @@ Run it!
alpha_opt <span style="color: #666666">=</span> sopt<span style="color: #666666">.</span>golden(f1d)
next_guess <span style="color: #666666">=</span> x <span style="color: #666666">+</span> alpha_opt <span style="color: #666666">*</span> s
guesses<span style="color: #666666">.</span>append(next_guess)
<span style="color: #008000; font-weight: bold">print</span>(next_guess)
<span style="color: #008000">print</span>(next_guess)
</pre></div>
<p>
What happened?
doc/pub/week39/html/week39-bs.html
+1 -1
View File
@@ -301,7 +301,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Sep 22, 2020</h4></center> <!-- date -->
<br>
<p>
doc/pub/week39/html/week39-reveal.html
+56 -50
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Sep 22, 2020</h4></center> <!-- date -->
<br>
<p>
@@ -165,11 +165,17 @@ MathJax.Hub.Config({
<p><li> Thursday: Repetition of Logistic regression equations and discussion of Gradient methods</li>
<p><li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
</ul>
<p>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapter 4</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf" target="_blank">Murphy sections 8.3 and 8.5</a>
</section>
<section>
<h2 id="___sec1">Thursday September 24 </h2>
<p>
"Overview Video, why do we care about gradient methods?":""
</section>
@@ -784,7 +790,7 @@ y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*
xb = np.c_[np.ones((m,<span style="color: #B452CD">1</span>)), x]
beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
<span style="color: #8B008B; font-weight: bold">print</span>(beta_linreg)
<span style="color: #658b00">print</span>(beta_linreg)
beta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
eta = <span style="color: #B452CD">0.1</span>
@@ -794,7 +800,7 @@ Niterations = <span style="color: #B452CD">1000</span>
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T.dot(xb.dot(beta)-y)
beta -= eta*gradients
<span style="color: #8B008B; font-weight: bold">print</span>(beta)
<span style="color: #658b00">print</span>(beta)
xnew = np.array([[<span style="color: #B452CD">0</span>],[<span style="color: #B452CD">2</span>]])
xbnew = np.c_[np.ones((<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)), xnew]
ypredict = xbnew.dot(beta)
@@ -828,10 +834,10 @@ y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*
xb = np.c_[np.ones((<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)), x]
beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
<span style="color: #8B008B; font-weight: bold">print</span>(beta_linreg)
sgdreg = SGDRegressor(n_iter = <span style="color: #B452CD">50</span>, penalty=<span style="color: #658b00">None</span>, eta0=<span style="color: #B452CD">0.1</span>)
<span style="color: #658b00">print</span>(beta_linreg)
sgdreg = SGDRegressor(n_iter = <span style="color: #B452CD">50</span>, penalty=<span style="color: #8B008B; font-weight: bold">None</span>, eta0=<span style="color: #B452CD">0.1</span>)
sgdreg.fit(x,y.ravel())
<span style="color: #8B008B; font-weight: bold">print</span>(sgdreg.intercept_, sgdreg.coef_)
<span style="color: #658b00">print</span>(sgdreg.intercept_, sgdreg.coef_)
</pre></div>
</section>
@@ -886,14 +892,14 @@ x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(m,<span style="color: #B452CD">1</span>)
xb = np.c_[np.ones((m,<span style="color: #B452CD">1</span>)), x]
XT_X = xb.T <span style="color: #a61717; background-color: #e3d2d2">@</span> xb
XT_X = xb.T @ xb
<span style="color: #228B22">#Ridge parameter lambda</span>
lmbda = <span style="color: #B452CD">0.001</span>
Id = lmbda* np.eye(XT_X.shape[<span style="color: #B452CD">0</span>])
beta_linreg = np.linalg.inv(XT_X+Id) <span style="color: #a61717; background-color: #e3d2d2">@</span> xb.T <span style="color: #a61717; background-color: #e3d2d2">@</span> y
<span style="color: #8B008B; font-weight: bold">print</span>(beta_linreg)
beta_linreg = np.linalg.inv(XT_X+Id) @ xb.T @ y
<span style="color: #658b00">print</span>(beta_linreg)
<span style="color: #228B22"># Start plain gradient descent</span>
beta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
@@ -901,12 +907,12 @@ eta = <span style="color: #B452CD">0.1</span>
Niterations = <span style="color: #B452CD">100</span>
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T <span style="color: #a61717; background-color: #e3d2d2">@</span> (xb <span style="color: #a61717; background-color: #e3d2d2">@</span> (beta)-y)+<span style="color: #B452CD">2</span>*lmbda*beta
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T @ (xb @ (beta)-y)+<span style="color: #B452CD">2</span>*lmbda*beta
beta -= eta*gradients
<span style="color: #8B008B; font-weight: bold">print</span>(beta)
ypredict = xb <span style="color: #a61717; background-color: #e3d2d2">@</span> beta
ypredict2 = xb <span style="color: #a61717; background-color: #e3d2d2">@</span> beta_linreg
<span style="color: #658b00">print</span>(beta)
ypredict = xb @ beta
ypredict2 = xb @ beta_linreg
plt.plot(x, ypredict, <span style="color: #CD5555">&quot;r-&quot;</span>)
plt.plot(x, ypredict2, <span style="color: #CD5555">&quot;b-&quot;</span>)
plt.plot(x, y ,<span style="color: #CD5555">&#39;ro&#39;</span>)
@@ -1122,7 +1128,7 @@ j = <span style="color: #B452CD">0</span>
gamma_j = step_length(t,t0,t1)
j += <span style="color: #B452CD">1</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;gamma_j after %d epochs: %g&quot;</span> % (n_epochs,gamma_j))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;gamma_j after %d epochs: %g&quot;</span> % (n_epochs,gamma_j))
</pre></div>
</section>
@@ -1146,12 +1152,12 @@ y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*
xb = np.c_[np.ones((m,<span style="color: #B452CD">1</span>)), x]
theta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Own inversion&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(theta_linreg)
sgdreg = SGDRegressor(max_iter = <span style="color: #B452CD">50</span>, penalty=<span style="color: #658b00">None</span>, eta0=<span style="color: #B452CD">0.1</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Own inversion&quot;</span>)
<span style="color: #658b00">print</span>(theta_linreg)
sgdreg = SGDRegressor(max_iter = <span style="color: #B452CD">50</span>, penalty=<span style="color: #8B008B; font-weight: bold">None</span>, eta0=<span style="color: #B452CD">0.1</span>)
sgdreg.fit(x,y.ravel())
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;sgdreg from scikit&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(sgdreg.intercept_, sgdreg.coef_)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;sgdreg from scikit&quot;</span>)
<span style="color: #658b00">print</span>(sgdreg.intercept_, sgdreg.coef_)
theta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
@@ -1160,10 +1166,10 @@ Niterations = <span style="color: #B452CD">1000</span>
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T <span style="color: #a61717; background-color: #e3d2d2">@</span> ((xb <span style="color: #a61717; background-color: #e3d2d2">@</span> theta)-y)
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T @ ((xb @ theta)-y)
theta -= eta*gradients
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;theta frm own gd&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(theta)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;theta frm own gd&quot;</span>)
<span style="color: #658b00">print</span>(theta)
xnew = np.array([[<span style="color: #B452CD">0</span>],[<span style="color: #B452CD">2</span>]])
xbnew = np.c_[np.ones((<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)), xnew]
@@ -1183,11 +1189,11 @@ theta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color
random_index = np.random.randint(m)
xi = xb[random_index:random_index+<span style="color: #B452CD">1</span>]
yi = y[random_index:random_index+<span style="color: #B452CD">1</span>]
gradients = <span style="color: #B452CD">2</span> * xi.T <span style="color: #a61717; background-color: #e3d2d2">@</span> ((xi <span style="color: #a61717; background-color: #e3d2d2">@</span> theta)-yi)
gradients = <span style="color: #B452CD">2</span> * xi.T @ ((xi @ theta)-yi)
eta = learning_schedule(epoch*m+i)
theta = theta - eta*gradients
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;theta from own sdg&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(theta)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;theta from own sdg&quot;</span>)
<span style="color: #658b00">print</span>(theta)
plt.plot(xnew, ypredict, <span style="color: #CD5555">&quot;r-&quot;</span>)
plt.plot(xnew, ypredict2, <span style="color: #CD5555">&quot;b-&quot;</span>)
@@ -1555,7 +1561,7 @@ plt.legend()
plt.show()
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The max absolute difference is: %g&quot;</span>%(np.max(np.abs(computed - analytic))))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The max absolute difference is: %g&quot;</span>%(np.max(np.abs(computed - analytic))))
</pre></div>
</section>
@@ -1585,11 +1591,11 @@ f1_grad = grad(f1)
a = <span style="color: #B452CD">1.0</span>
<span style="color: #228B22"># See the evaluated gradient at a using autograd:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The gradient of f1 evaluated at a = %g using autograd is: %g&quot;</span>%(a,f1_grad(a)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The gradient of f1 evaluated at a = %g using autograd is: %g&quot;</span>%(a,f1_grad(a)))
<span style="color: #228B22"># Compare with the analytical derivative, that is f1&#39;(x) = 3*x**2 </span>
grad_analytical = <span style="color: #B452CD">3</span>*a**<span style="color: #B452CD">2</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The gradient of f1 evaluated at a = %g by finding the analytic expression is: %g&quot;</span>%(a,grad_analytical))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The gradient of f1 evaluated at a = %g by finding the analytic expression is: %g&quot;</span>%(a,grad_analytical))
</pre></div>
</section>
@@ -1619,8 +1625,8 @@ f2_grad_x2 = grad(f2,<span style="color: #B452CD">1</span>)
x1 = <span style="color: #B452CD">1.0</span>
x2 = <span style="color: #B452CD">3.0</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Evaluating at x1 = %g, x2 = %g&quot;</span>%(x1,x2))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;-&quot;</span>*<span style="color: #B452CD">30</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Evaluating at x1 = %g, x2 = %g&quot;</span>%(x1,x2))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;-&quot;</span>*<span style="color: #B452CD">30</span>)
<span style="color: #228B22"># Compare with the analytical derivatives:</span>
@@ -1631,13 +1637,13 @@ f2_grad_x1_analytical = <span style="color: #B452CD">9</span>*x1**<span style="c
f2_grad_x2_analytical = x1 - <span style="color: #B452CD">5</span>
<span style="color: #228B22"># See the evaluated derivations:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The derivative of f2 w.r.t x1: %g&quot;</span>%( f2_grad_x1(x1,x2) ))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f2 w.r.t x1: %g&quot;</span>%( f2_grad_x1(x1,x2) ))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The derivative of f2 w.r.t x1: %g&quot;</span>%( f2_grad_x1(x1,x2) ))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f2 w.r.t x1: %g&quot;</span>%( f2_grad_x1(x1,x2) ))
<span style="color: #8B008B; font-weight: bold">print</span>()
<span style="color: #658b00">print</span>()
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The derivative of f2 w.r.t x2: %g&quot;</span>%( f2_grad_x2(x1,x2) ))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f2 w.r.t x2: %g&quot;</span>%( f2_grad_x2(x1,x2) ))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The derivative of f2 w.r.t x2: %g&quot;</span>%( f2_grad_x2(x1,x2) ))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f2 w.r.t x2: %g&quot;</span>%( f2_grad_x2(x1,x2) ))
</pre></div>
<p>
Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
@@ -1660,13 +1666,13 @@ f3_grad = grad(f3)
x = np.linspace(<span style="color: #B452CD">0</span>,<span style="color: #B452CD">4</span>,<span style="color: #B452CD">5</span>)
<span style="color: #228B22"># Print the computed gradient:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed gradient of f3 is: &quot;</span>, f3_grad(x))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed gradient of f3 is: &quot;</span>, f3_grad(x))
<span style="color: #228B22"># The analytical gradient is: (2, 3, 5, 7, 22*x[4])</span>
f3_grad_analytical = np.array([<span style="color: #B452CD">2</span>, <span style="color: #B452CD">3</span>, <span style="color: #B452CD">5</span>, <span style="color: #B452CD">7</span>, <span style="color: #B452CD">22</span>*x[<span style="color: #B452CD">4</span>]])
<span style="color: #228B22"># Print the analytical gradient:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical gradient of f3 is: &quot;</span>, f3_grad_analytical)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical gradient of f3 is: &quot;</span>, f3_grad_analytical)
</pre></div>
<p>
Note that in this case, when sending an array as input argument, the
@@ -1694,13 +1700,13 @@ f4_grad = grad(f4)
x = <span style="color: #B452CD">2.7</span>
<span style="color: #228B22"># Print the computed derivative:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f4 at x = %g is: %g&quot;</span>%(x,f4_grad(x)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f4 at x = %g is: %g&quot;</span>%(x,f4_grad(x)))
<span style="color: #228B22"># The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi</span>
f4_grad_analytical = x/np.sqrt(<span style="color: #B452CD">1</span> + x**<span style="color: #B452CD">2</span>) + np.exp(x) + np.cos(<span style="color: #B452CD">2</span>*np.pi*x)*<span style="color: #B452CD">2</span>*np.pi
<span style="color: #228B22"># Print the analytical gradient:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical gradient of f4 at x = %g is: %g&quot;</span>%(x,f4_grad_analytical))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical gradient of f4 at x = %g is: %g&quot;</span>%(x,f4_grad_analytical))
</pre></div>
</section>
@@ -1724,7 +1730,7 @@ f5_grad = grad(f5)
x = <span style="color: #B452CD">2.7</span>
<span style="color: #228B22"># Print the computed derivative:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f5 at x = %g is: %g&quot;</span>%(x,f5_grad(x)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f5 at x = %g is: %g&quot;</span>%(x,f5_grad(x)))
</pre></div>
</section>
@@ -1757,8 +1763,8 @@ f6_while_grad = grad(f6_while)
x = <span style="color: #B452CD">0.5</span>
<span style="color: #228B22"># Print the computed derivaties of f6_for and f6_while</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f6_for at x = %g is: %g&quot;</span>%(x,f6_for_grad(x)))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f6_while at x = %g is: %g&quot;</span>%(x,f6_while_grad(x)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f6_for at x = %g is: %g&quot;</span>%(x,f6_for_grad(x)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f6_while at x = %g is: %g&quot;</span>%(x,f6_while_grad(x)))
</pre></div>
<p>
@@ -1771,7 +1777,7 @@ f6_grad_analytical = <span style="color: #B452CD">0</span>
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #B452CD">10</span>):
f6_grad_analytical += i*x**(i-<span style="color: #B452CD">1</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f6 at x = %g is: %g&quot;</span>%(x,f6_grad_analytical))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f6 at x = %g is: %g&quot;</span>%(x,f6_grad_analytical))
</pre></div>
</section>
@@ -1794,7 +1800,7 @@ f7_grad = grad(f7)
n = <span style="color: #B452CD">2.0</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f7 at n = %d is: %g&quot;</span>%(n,f7_grad(n)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f7 at n = %d is: %g&quot;</span>%(n,f7_grad(n)))
<span style="color: #228B22"># The function f7 is an implementation of the factorial of n.</span>
<span style="color: #228B22"># By using the product rule, one can find that the derivative is:</span>
@@ -1807,7 +1813,7 @@ f7_grad_analytical = <span style="color: #B452CD">0</span>
tmp *= (n - k)
f7_grad_analytical += tmp
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f7 at n = %d is: %g&quot;</span>%(n,f7_grad_analytical))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f7 at n = %d is: %g&quot;</span>%(n,f7_grad_analytical))
</pre></div>
<p>
Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
@@ -1833,7 +1839,7 @@ f8_grad = grad(f8)
x = <span style="color: #B452CD">8.4</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The derivative of f8 is:&quot;</span>,f8_grad(x))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The derivative of f8 is:&quot;</span>,f8_grad(x))
</pre></div>
<p>
Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
@@ -1855,7 +1861,7 @@ f9_grad = grad(f9)
x = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0.0</span>])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The derivative of f9 is:&quot;</span>,f9_grad(x))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The derivative of f9 is:&quot;</span>,f9_grad(x))
</pre></div>
<p>
Here we are told that the 'dot' function does not belong to Autograd's
@@ -1875,7 +1881,7 @@ f9_alternative_grad = grad(f9_alternative)
x = np.array([<span style="color: #B452CD">3.0</span>,<span style="color: #B452CD">0.0</span>])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The gradient of f9 is:&quot;</span>,f9_alternative_grad(x))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The gradient of f9 is:&quot;</span>,f9_alternative_grad(x))
<span style="color: #228B22"># The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively</span>
<span style="color: #228B22"># w.r.t x is (b_1, b_2).</span>
@@ -2168,7 +2174,7 @@ $$
fig = pt.figure()
ax = fig.gca(projection=<span style="color: #CD5555">&quot;3d&quot;</span>)
xmesh, ymesh = np.mgrid[-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50j</span>,-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50j</span>]
xmesh, ymesh = np.mgrid[-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50</span>j,-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50</span>j]
fmesh = f(np.array([xmesh, ymesh]))
ax.plot_surface(xmesh, ymesh, fmesh)
</pre></div>
@@ -2200,7 +2206,7 @@ Run it!
alpha_opt = sopt.golden(f1d)
next_guess = x + alpha_opt * s
guesses.append(next_guess)
<span style="color: #8B008B; font-weight: bold">print</span>(next_guess)
<span style="color: #658b00">print</span>(next_guess)
</pre></div>
<p>
What happened?
doc/pub/week39/html/week39-solarized.html
+56 -50
View File
@@ -216,7 +216,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Sep 22, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -228,10 +228,16 @@ MathJax.Hub.Config({
<li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
</ul>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapter 4</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf" target="_blank">Murphy sections 8.3 and 8.5</a>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Thursday September 24 </h2>
<p>
"Overview Video, why do we care about gradient methods?":""
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -795,7 +801,7 @@ y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*
xb = np.c_[np.ones((m,<span style="color: #B452CD">1</span>)), x]
beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
<span style="color: #8B008B; font-weight: bold">print</span>(beta_linreg)
<span style="color: #658b00">print</span>(beta_linreg)
beta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
eta = <span style="color: #B452CD">0.1</span>
@@ -805,7 +811,7 @@ Niterations = <span style="color: #B452CD">1000</span>
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T.dot(xb.dot(beta)-y)
beta -= eta*gradients
<span style="color: #8B008B; font-weight: bold">print</span>(beta)
<span style="color: #658b00">print</span>(beta)
xnew = np.array([[<span style="color: #B452CD">0</span>],[<span style="color: #B452CD">2</span>]])
xbnew = np.c_[np.ones((<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)), xnew]
ypredict = xbnew.dot(beta)
@@ -838,10 +844,10 @@ y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*
xb = np.c_[np.ones((<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)), x]
beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
<span style="color: #8B008B; font-weight: bold">print</span>(beta_linreg)
sgdreg = SGDRegressor(n_iter = <span style="color: #B452CD">50</span>, penalty=<span style="color: #658b00">None</span>, eta0=<span style="color: #B452CD">0.1</span>)
<span style="color: #658b00">print</span>(beta_linreg)
sgdreg = SGDRegressor(n_iter = <span style="color: #B452CD">50</span>, penalty=<span style="color: #8B008B; font-weight: bold">None</span>, eta0=<span style="color: #B452CD">0.1</span>)
sgdreg.fit(x,y.ravel())
<span style="color: #8B008B; font-weight: bold">print</span>(sgdreg.intercept_, sgdreg.coef_)
<span style="color: #658b00">print</span>(sgdreg.intercept_, sgdreg.coef_)
</pre></div>
<p>
<!-- !split -->
@@ -889,14 +895,14 @@ x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(m,<span style="color: #B452CD">1</span>)
xb = np.c_[np.ones((m,<span style="color: #B452CD">1</span>)), x]
XT_X = xb.T <span style="color: #a61717; background-color: #e3d2d2">@</span> xb
XT_X = xb.T @ xb
<span style="color: #228B22">#Ridge parameter lambda</span>
lmbda = <span style="color: #B452CD">0.001</span>
Id = lmbda* np.eye(XT_X.shape[<span style="color: #B452CD">0</span>])
beta_linreg = np.linalg.inv(XT_X+Id) <span style="color: #a61717; background-color: #e3d2d2">@</span> xb.T <span style="color: #a61717; background-color: #e3d2d2">@</span> y
<span style="color: #8B008B; font-weight: bold">print</span>(beta_linreg)
beta_linreg = np.linalg.inv(XT_X+Id) @ xb.T @ y
<span style="color: #658b00">print</span>(beta_linreg)
<span style="color: #228B22"># Start plain gradient descent</span>
beta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
@@ -904,12 +910,12 @@ eta = <span style="color: #B452CD">0.1</span>
Niterations = <span style="color: #B452CD">100</span>
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T <span style="color: #a61717; background-color: #e3d2d2">@</span> (xb <span style="color: #a61717; background-color: #e3d2d2">@</span> (beta)-y)+<span style="color: #B452CD">2</span>*lmbda*beta
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T @ (xb @ (beta)-y)+<span style="color: #B452CD">2</span>*lmbda*beta
beta -= eta*gradients
<span style="color: #8B008B; font-weight: bold">print</span>(beta)
ypredict = xb <span style="color: #a61717; background-color: #e3d2d2">@</span> beta
ypredict2 = xb <span style="color: #a61717; background-color: #e3d2d2">@</span> beta_linreg
<span style="color: #658b00">print</span>(beta)
ypredict = xb @ beta
ypredict2 = xb @ beta_linreg
plt.plot(x, ypredict, <span style="color: #CD5555">&quot;r-&quot;</span>)
plt.plot(x, ypredict2, <span style="color: #CD5555">&quot;b-&quot;</span>)
plt.plot(x, y ,<span style="color: #CD5555">&#39;ro&#39;</span>)
@@ -1113,7 +1119,7 @@ j = <span style="color: #B452CD">0</span>
gamma_j = step_length(t,t0,t1)
j += <span style="color: #B452CD">1</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;gamma_j after %d epochs: %g&quot;</span> % (n_epochs,gamma_j))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;gamma_j after %d epochs: %g&quot;</span> % (n_epochs,gamma_j))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1136,12 +1142,12 @@ y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*
xb = np.c_[np.ones((m,<span style="color: #B452CD">1</span>)), x]
theta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Own inversion&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(theta_linreg)
sgdreg = SGDRegressor(max_iter = <span style="color: #B452CD">50</span>, penalty=<span style="color: #658b00">None</span>, eta0=<span style="color: #B452CD">0.1</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Own inversion&quot;</span>)
<span style="color: #658b00">print</span>(theta_linreg)
sgdreg = SGDRegressor(max_iter = <span style="color: #B452CD">50</span>, penalty=<span style="color: #8B008B; font-weight: bold">None</span>, eta0=<span style="color: #B452CD">0.1</span>)
sgdreg.fit(x,y.ravel())
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;sgdreg from scikit&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(sgdreg.intercept_, sgdreg.coef_)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;sgdreg from scikit&quot;</span>)
<span style="color: #658b00">print</span>(sgdreg.intercept_, sgdreg.coef_)
theta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
@@ -1150,10 +1156,10 @@ Niterations = <span style="color: #B452CD">1000</span>
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T <span style="color: #a61717; background-color: #e3d2d2">@</span> ((xb <span style="color: #a61717; background-color: #e3d2d2">@</span> theta)-y)
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T @ ((xb @ theta)-y)
theta -= eta*gradients
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;theta frm own gd&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(theta)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;theta frm own gd&quot;</span>)
<span style="color: #658b00">print</span>(theta)
xnew = np.array([[<span style="color: #B452CD">0</span>],[<span style="color: #B452CD">2</span>]])
xbnew = np.c_[np.ones((<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)), xnew]
@@ -1173,11 +1179,11 @@ theta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color
random_index = np.random.randint(m)
xi = xb[random_index:random_index+<span style="color: #B452CD">1</span>]
yi = y[random_index:random_index+<span style="color: #B452CD">1</span>]
gradients = <span style="color: #B452CD">2</span> * xi.T <span style="color: #a61717; background-color: #e3d2d2">@</span> ((xi <span style="color: #a61717; background-color: #e3d2d2">@</span> theta)-yi)
gradients = <span style="color: #B452CD">2</span> * xi.T @ ((xi @ theta)-yi)
eta = learning_schedule(epoch*m+i)
theta = theta - eta*gradients
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;theta from own sdg&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(theta)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;theta from own sdg&quot;</span>)
<span style="color: #658b00">print</span>(theta)
plt.plot(xnew, ypredict, <span style="color: #CD5555">&quot;r-&quot;</span>)
plt.plot(xnew, ypredict2, <span style="color: #CD5555">&quot;b-&quot;</span>)
@@ -1519,7 +1525,7 @@ plt.legend()
plt.show()
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The max absolute difference is: %g&quot;</span>%(np.max(np.abs(computed - analytic))))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The max absolute difference is: %g&quot;</span>%(np.max(np.abs(computed - analytic))))
</pre></div>
<p>
<!-- !split -->
@@ -1548,11 +1554,11 @@ f1_grad = grad(f1)
a = <span style="color: #B452CD">1.0</span>
<span style="color: #228B22"># See the evaluated gradient at a using autograd:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The gradient of f1 evaluated at a = %g using autograd is: %g&quot;</span>%(a,f1_grad(a)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The gradient of f1 evaluated at a = %g using autograd is: %g&quot;</span>%(a,f1_grad(a)))
<span style="color: #228B22"># Compare with the analytical derivative, that is f1&#39;(x) = 3*x**2 </span>
grad_analytical = <span style="color: #B452CD">3</span>*a**<span style="color: #B452CD">2</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The gradient of f1 evaluated at a = %g by finding the analytic expression is: %g&quot;</span>%(a,grad_analytical))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The gradient of f1 evaluated at a = %g by finding the analytic expression is: %g&quot;</span>%(a,grad_analytical))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1581,8 +1587,8 @@ f2_grad_x2 = grad(f2,<span style="color: #B452CD">1</span>)
x1 = <span style="color: #B452CD">1.0</span>
x2 = <span style="color: #B452CD">3.0</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Evaluating at x1 = %g, x2 = %g&quot;</span>%(x1,x2))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;-&quot;</span>*<span style="color: #B452CD">30</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Evaluating at x1 = %g, x2 = %g&quot;</span>%(x1,x2))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;-&quot;</span>*<span style="color: #B452CD">30</span>)
<span style="color: #228B22"># Compare with the analytical derivatives:</span>
@@ -1593,13 +1599,13 @@ f2_grad_x1_analytical = <span style="color: #B452CD">9</span>*x1**<span style="c
f2_grad_x2_analytical = x1 - <span style="color: #B452CD">5</span>
<span style="color: #228B22"># See the evaluated derivations:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The derivative of f2 w.r.t x1: %g&quot;</span>%( f2_grad_x1(x1,x2) ))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f2 w.r.t x1: %g&quot;</span>%( f2_grad_x1(x1,x2) ))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The derivative of f2 w.r.t x1: %g&quot;</span>%( f2_grad_x1(x1,x2) ))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f2 w.r.t x1: %g&quot;</span>%( f2_grad_x1(x1,x2) ))
<span style="color: #8B008B; font-weight: bold">print</span>()
<span style="color: #658b00">print</span>()
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The derivative of f2 w.r.t x2: %g&quot;</span>%( f2_grad_x2(x1,x2) ))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f2 w.r.t x2: %g&quot;</span>%( f2_grad_x2(x1,x2) ))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The derivative of f2 w.r.t x2: %g&quot;</span>%( f2_grad_x2(x1,x2) ))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f2 w.r.t x2: %g&quot;</span>%( f2_grad_x2(x1,x2) ))
</pre></div>
<p>
Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
@@ -1622,13 +1628,13 @@ f3_grad = grad(f3)
x = np.linspace(<span style="color: #B452CD">0</span>,<span style="color: #B452CD">4</span>,<span style="color: #B452CD">5</span>)
<span style="color: #228B22"># Print the computed gradient:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed gradient of f3 is: &quot;</span>, f3_grad(x))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed gradient of f3 is: &quot;</span>, f3_grad(x))
<span style="color: #228B22"># The analytical gradient is: (2, 3, 5, 7, 22*x[4])</span>
f3_grad_analytical = np.array([<span style="color: #B452CD">2</span>, <span style="color: #B452CD">3</span>, <span style="color: #B452CD">5</span>, <span style="color: #B452CD">7</span>, <span style="color: #B452CD">22</span>*x[<span style="color: #B452CD">4</span>]])
<span style="color: #228B22"># Print the analytical gradient:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical gradient of f3 is: &quot;</span>, f3_grad_analytical)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical gradient of f3 is: &quot;</span>, f3_grad_analytical)
</pre></div>
<p>
Note that in this case, when sending an array as input argument, the
@@ -1656,13 +1662,13 @@ f4_grad = grad(f4)
x = <span style="color: #B452CD">2.7</span>
<span style="color: #228B22"># Print the computed derivative:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f4 at x = %g is: %g&quot;</span>%(x,f4_grad(x)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f4 at x = %g is: %g&quot;</span>%(x,f4_grad(x)))
<span style="color: #228B22"># The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi</span>
f4_grad_analytical = x/np.sqrt(<span style="color: #B452CD">1</span> + x**<span style="color: #B452CD">2</span>) + np.exp(x) + np.cos(<span style="color: #B452CD">2</span>*np.pi*x)*<span style="color: #B452CD">2</span>*np.pi
<span style="color: #228B22"># Print the analytical gradient:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical gradient of f4 at x = %g is: %g&quot;</span>%(x,f4_grad_analytical))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical gradient of f4 at x = %g is: %g&quot;</span>%(x,f4_grad_analytical))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1685,7 +1691,7 @@ f5_grad = grad(f5)
x = <span style="color: #B452CD">2.7</span>
<span style="color: #228B22"># Print the computed derivative:</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f5 at x = %g is: %g&quot;</span>%(x,f5_grad(x)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f5 at x = %g is: %g&quot;</span>%(x,f5_grad(x)))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1717,8 +1723,8 @@ f6_while_grad = grad(f6_while)
x = <span style="color: #B452CD">0.5</span>
<span style="color: #228B22"># Print the computed derivaties of f6_for and f6_while</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f6_for at x = %g is: %g&quot;</span>%(x,f6_for_grad(x)))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f6_while at x = %g is: %g&quot;</span>%(x,f6_while_grad(x)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f6_for at x = %g is: %g&quot;</span>%(x,f6_for_grad(x)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f6_while at x = %g is: %g&quot;</span>%(x,f6_while_grad(x)))
</pre></div>
<p>
@@ -1731,7 +1737,7 @@ f6_grad_analytical = <span style="color: #B452CD">0</span>
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #B452CD">10</span>):
f6_grad_analytical += i*x**(i-<span style="color: #B452CD">1</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f6 at x = %g is: %g&quot;</span>%(x,f6_grad_analytical))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f6 at x = %g is: %g&quot;</span>%(x,f6_grad_analytical))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1753,7 +1759,7 @@ f7_grad = grad(f7)
n = <span style="color: #B452CD">2.0</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f7 at n = %d is: %g&quot;</span>%(n,f7_grad(n)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The computed derivative of f7 at n = %d is: %g&quot;</span>%(n,f7_grad(n)))
<span style="color: #228B22"># The function f7 is an implementation of the factorial of n.</span>
<span style="color: #228B22"># By using the product rule, one can find that the derivative is:</span>
@@ -1766,7 +1772,7 @@ f7_grad_analytical = <span style="color: #B452CD">0</span>
tmp *= (n - k)
f7_grad_analytical += tmp
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f7 at n = %d is: %g&quot;</span>%(n,f7_grad_analytical))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The analytical derivative of f7 at n = %d is: %g&quot;</span>%(n,f7_grad_analytical))
</pre></div>
<p>
Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
@@ -1792,7 +1798,7 @@ f8_grad = grad(f8)
x = <span style="color: #B452CD">8.4</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The derivative of f8 is:&quot;</span>,f8_grad(x))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The derivative of f8 is:&quot;</span>,f8_grad(x))
</pre></div>
<p>
Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
@@ -1814,7 +1820,7 @@ f9_grad = grad(f9)
x = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0.0</span>])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The derivative of f9 is:&quot;</span>,f9_grad(x))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The derivative of f9 is:&quot;</span>,f9_grad(x))
</pre></div>
<p>
Here we are told that the 'dot' function does not belong to Autograd's
@@ -1834,7 +1840,7 @@ f9_alternative_grad = grad(f9_alternative)
x = np.array([<span style="color: #B452CD">3.0</span>,<span style="color: #B452CD">0.0</span>])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;The gradient of f9 is:&quot;</span>,f9_alternative_grad(x))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;The gradient of f9 is:&quot;</span>,f9_alternative_grad(x))
<span style="color: #228B22"># The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively</span>
<span style="color: #228B22"># w.r.t x is (b_1, b_2).</span>
@@ -2107,7 +2113,7 @@ $$
fig = pt.figure()
ax = fig.gca(projection=<span style="color: #CD5555">&quot;3d&quot;</span>)
xmesh, ymesh = np.mgrid[-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50j</span>,-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50j</span>]
xmesh, ymesh = np.mgrid[-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50</span>j,-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50</span>j]
fmesh = f(np.array([xmesh, ymesh]))
ax.plot_surface(xmesh, ymesh, fmesh)
</pre></div>
@@ -2139,7 +2145,7 @@ Run it!
alpha_opt = sopt.golden(f1d)
next_guess = x + alpha_opt * s
guesses.append(next_guess)
<span style="color: #8B008B; font-weight: bold">print</span>(next_guess)
<span style="color: #658b00">print</span>(next_guess)
</pre></div>
<p>
What happened?
doc/pub/week39/html/week39.html
+56 -50
View File
@@ -221,7 +221,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Sep 22, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -233,10 +233,16 @@ MathJax.Hub.Config({
<li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
</ul>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapter 4</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf" target="_blank">Murphy sections 8.3 and 8.5</a>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Thursday September 24 </h2>
<p>
"Overview Video, why do we care about gradient methods?":""
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -800,7 +806,7 @@ y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((m,<span style="color: #666666">1</span>)), x]
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb))<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>T)<span style="color: #666666">.</span>dot(y)
<span style="color: #008000; font-weight: bold">print</span>(beta_linreg)
<span style="color: #008000">print</span>(beta_linreg)
beta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)
eta <span style="color: #666666">=</span> <span style="color: #666666">0.1</span>
@@ -810,7 +816,7 @@ Niterations <span style="color: #666666">=</span> <span style="color: #666666">1
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>dot(beta)<span style="color: #666666">-</span>y)
beta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradients
<span style="color: #008000; font-weight: bold">print</span>(beta)
<span style="color: #008000">print</span>(beta)
xnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">0</span>],[<span style="color: #666666">2</span>]])
xbnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)), xnew]
ypredict <span style="color: #666666">=</span> xbnew<span style="color: #666666">.</span>dot(beta)
@@ -843,10 +849,10 @@ y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)), x]
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb))<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>T)<span style="color: #666666">.</span>dot(y)
<span style="color: #008000; font-weight: bold">print</span>(beta_linreg)
sgdreg <span style="color: #666666">=</span> SGDRegressor(n_iter <span style="color: #666666">=</span> <span style="color: #666666">50</span>, penalty<span style="color: #666666">=</span><span style="color: #008000">None</span>, eta0<span style="color: #666666">=0.1</span>)
<span style="color: #008000">print</span>(beta_linreg)
sgdreg <span style="color: #666666">=</span> SGDRegressor(n_iter <span style="color: #666666">=</span> <span style="color: #666666">50</span>, penalty<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>, eta0<span style="color: #666666">=0.1</span>)
sgdreg<span style="color: #666666">.</span>fit(x,y<span style="color: #666666">.</span>ravel())
<span style="color: #008000; font-weight: bold">print</span>(sgdreg<span style="color: #666666">.</span>intercept_, sgdreg<span style="color: #666666">.</span>coef_)
<span style="color: #008000">print</span>(sgdreg<span style="color: #666666">.</span>intercept_, sgdreg<span style="color: #666666">.</span>coef_)
</pre></div>
<p>
<!-- !split -->
@@ -894,14 +900,14 @@ x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(m,<span style="color: #666666">1</span>)
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((m,<span style="color: #666666">1</span>)), x]
XT_X <span style="color: #666666">=</span> xb<span style="color: #666666">.</span>T @ xb
XT_X <span style="color: #666666">=</span> xb<span style="color: #666666">.</span>T <span style="color: #666666">@</span> xb
<span style="color: #408080; font-style: italic">#Ridge parameter lambda</span>
lmbda <span style="color: #666666">=</span> <span style="color: #666666">0.001</span>
Id <span style="color: #666666">=</span> lmbda<span style="color: #666666">*</span> np<span style="color: #666666">.</span>eye(XT_X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>])
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(XT_X<span style="color: #666666">+</span>Id) @ xb<span style="color: #666666">.</span>T @ y
<span style="color: #008000; font-weight: bold">print</span>(beta_linreg)
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(XT_X<span style="color: #666666">+</span>Id) <span style="color: #666666">@</span> xb<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y
<span style="color: #008000">print</span>(beta_linreg)
<span style="color: #408080; font-style: italic"># Start plain gradient descent</span>
beta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)
@@ -909,12 +915,12 @@ eta <span style="color: #666666">=</span> <span style="color: #666666">0.1</span
Niterations <span style="color: #666666">=</span> <span style="color: #666666">100</span>
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(Niterations):
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T @ (xb @ (beta)<span style="color: #666666">-</span>y)<span style="color: #666666">+2*</span>lmbda<span style="color: #666666">*</span>beta
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T <span style="color: #666666">@</span> (xb <span style="color: #666666">@</span> (beta)<span style="color: #666666">-</span>y)<span style="color: #666666">+2*</span>lmbda<span style="color: #666666">*</span>beta
beta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradients
<span style="color: #008000; font-weight: bold">print</span>(beta)
ypredict <span style="color: #666666">=</span> xb @ beta
ypredict2 <span style="color: #666666">=</span> xb @ beta_linreg
<span style="color: #008000">print</span>(beta)
ypredict <span style="color: #666666">=</span> xb <span style="color: #666666">@</span> beta
ypredict2 <span style="color: #666666">=</span> xb <span style="color: #666666">@</span> beta_linreg
plt<span style="color: #666666">.</span>plot(x, ypredict, <span style="color: #BA2121">&quot;r-&quot;</span>)
plt<span style="color: #666666">.</span>plot(x, ypredict2, <span style="color: #BA2121">&quot;b-&quot;</span>)
plt<span style="color: #666666">.</span>plot(x, y ,<span style="color: #BA2121">&#39;ro&#39;</span>)
@@ -1118,7 +1124,7 @@ j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
gamma_j <span style="color: #666666">=</span> step_length(t,t0,t1)
j <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;gamma_j after </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> epochs: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> (n_epochs,gamma_j))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;gamma_j after </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> epochs: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> (n_epochs,gamma_j))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1141,12 +1147,12 @@ y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((m,<span style="color: #666666">1</span>)), x]
theta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb))<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>T)<span style="color: #666666">.</span>dot(y)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Own inversion&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(theta_linreg)
sgdreg <span style="color: #666666">=</span> SGDRegressor(max_iter <span style="color: #666666">=</span> <span style="color: #666666">50</span>, penalty<span style="color: #666666">=</span><span style="color: #008000">None</span>, eta0<span style="color: #666666">=0.1</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Own inversion&quot;</span>)
<span style="color: #008000">print</span>(theta_linreg)
sgdreg <span style="color: #666666">=</span> SGDRegressor(max_iter <span style="color: #666666">=</span> <span style="color: #666666">50</span>, penalty<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>, eta0<span style="color: #666666">=0.1</span>)
sgdreg<span style="color: #666666">.</span>fit(x,y<span style="color: #666666">.</span>ravel())
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;sgdreg from scikit&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(sgdreg<span style="color: #666666">.</span>intercept_, sgdreg<span style="color: #666666">.</span>coef_)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;sgdreg from scikit&quot;</span>)
<span style="color: #008000">print</span>(sgdreg<span style="color: #666666">.</span>intercept_, sgdreg<span style="color: #666666">.</span>coef_)
theta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)
@@ -1155,10 +1161,10 @@ Niterations <span style="color: #666666">=</span> <span style="color: #666666">1
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(Niterations):
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T @ ((xb @ theta)<span style="color: #666666">-</span>y)
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T <span style="color: #666666">@</span> ((xb <span style="color: #666666">@</span> theta)<span style="color: #666666">-</span>y)
theta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradients
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;theta frm own gd&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(theta)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;theta frm own gd&quot;</span>)
<span style="color: #008000">print</span>(theta)
xnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">0</span>],[<span style="color: #666666">2</span>]])
xbnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)), xnew]
@@ -1178,11 +1184,11 @@ theta <span style="color: #666666">=</span> np<span style="color: #666666">.</sp
random_index <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randint(m)
xi <span style="color: #666666">=</span> xb[random_index:random_index<span style="color: #666666">+1</span>]
yi <span style="color: #666666">=</span> y[random_index:random_index<span style="color: #666666">+1</span>]
gradients <span style="color: #666666">=</span> <span style="color: #666666">2</span> <span style="color: #666666">*</span> xi<span style="color: #666666">.</span>T @ ((xi @ theta)<span style="color: #666666">-</span>yi)
gradients <span style="color: #666666">=</span> <span style="color: #666666">2</span> <span style="color: #666666">*</span> xi<span style="color: #666666">.</span>T <span style="color: #666666">@</span> ((xi <span style="color: #666666">@</span> theta)<span style="color: #666666">-</span>yi)
eta <span style="color: #666666">=</span> learning_schedule(epoch<span style="color: #666666">*</span>m<span style="color: #666666">+</span>i)
theta <span style="color: #666666">=</span> theta <span style="color: #666666">-</span> eta<span style="color: #666666">*</span>gradients
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;theta from own sdg&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(theta)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;theta from own sdg&quot;</span>)
<span style="color: #008000">print</span>(theta)
plt<span style="color: #666666">.</span>plot(xnew, ypredict, <span style="color: #BA2121">&quot;r-&quot;</span>)
plt<span style="color: #666666">.</span>plot(xnew, ypredict2, <span style="color: #BA2121">&quot;b-&quot;</span>)
@@ -1524,7 +1530,7 @@ plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The max absolute difference is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(np<span style="color: #666666">.</span>max(np<span style="color: #666666">.</span>abs(computed <span style="color: #666666">-</span> analytic))))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The max absolute difference is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(np<span style="color: #666666">.</span>max(np<span style="color: #666666">.</span>abs(computed <span style="color: #666666">-</span> analytic))))
</pre></div>
<p>
<!-- !split -->
@@ -1553,11 +1559,11 @@ f1_grad <span style="color: #666666">=</span> grad(f1)
a <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
<span style="color: #408080; font-style: italic"># See the evaluated gradient at a using autograd:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> using autograd is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,f1_grad(a)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> using autograd is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,f1_grad(a)))
<span style="color: #408080; font-style: italic"># Compare with the analytical derivative, that is f1&#39;(x) = 3*x**2 </span>
grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">3*</span>a<span style="color: #666666">**2</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> by finding the analytic expression is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,grad_analytical))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> by finding the analytic expression is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,grad_analytical))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1586,8 +1592,8 @@ f2_grad_x2 <span style="color: #666666">=</span> grad(f2,<span style="color: #66
x1 <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
x2 <span style="color: #666666">=</span> <span style="color: #666666">3.0</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Evaluating at x1 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">, x2 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x1,x2))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;-&quot;</span><span style="color: #666666">*30</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Evaluating at x1 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">, x2 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x1,x2))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;-&quot;</span><span style="color: #666666">*30</span>)
<span style="color: #408080; font-style: italic"># Compare with the analytical derivatives:</span>
@@ -1598,13 +1604,13 @@ f2_grad_x1_analytical <span style="color: #666666">=</span> <span style="color:
f2_grad_x2_analytical <span style="color: #666666">=</span> x1 <span style="color: #666666">-</span> <span style="color: #666666">5</span>
<span style="color: #408080; font-style: italic"># See the evaluated derivations:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000; font-weight: bold">print</span>()
<span style="color: #008000">print</span>()
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
</pre></div>
<p>
Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
@@ -1627,13 +1633,13 @@ f3_grad <span style="color: #666666">=</span> grad(f3)
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">0</span>,<span style="color: #666666">4</span>,<span style="color: #666666">5</span>)
<span style="color: #408080; font-style: italic"># Print the computed gradient:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed gradient of f3 is: &quot;</span>, f3_grad(x))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed gradient of f3 is: &quot;</span>, f3_grad(x))
<span style="color: #408080; font-style: italic"># The analytical gradient is: (2, 3, 5, 7, 22*x[4])</span>
f3_grad_analytical <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">2</span>, <span style="color: #666666">3</span>, <span style="color: #666666">5</span>, <span style="color: #666666">7</span>, <span style="color: #666666">22*</span>x[<span style="color: #666666">4</span>]])
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f3 is: &quot;</span>, f3_grad_analytical)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f3 is: &quot;</span>, f3_grad_analytical)
</pre></div>
<p>
Note that in this case, when sending an array as input argument, the
@@ -1661,13 +1667,13 @@ f4_grad <span style="color: #666666">=</span> grad(f4)
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad(x)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad(x)))
<span style="color: #408080; font-style: italic"># The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi</span>
f4_grad_analytical <span style="color: #666666">=</span> x<span style="color: #666666">/</span>np<span style="color: #666666">.</span>sqrt(<span style="color: #666666">1</span> <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(x) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>cos(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x)<span style="color: #666666">*2*</span>np<span style="color: #666666">.</span>pi
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad_analytical))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad_analytical))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1690,7 +1696,7 @@ f5_grad <span style="color: #666666">=</span> grad(f5)
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f5 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f5_grad(x)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f5 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f5_grad(x)))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1722,8 +1728,8 @@ f6_while_grad <span style="color: #666666">=</span> grad(f6_while)
x <span style="color: #666666">=</span> <span style="color: #666666">0.5</span>
<span style="color: #408080; font-style: italic"># Print the computed derivaties of f6_for and f6_while</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_for at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_for_grad(x)))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_while at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_while_grad(x)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_for at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_for_grad(x)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_while at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_while_grad(x)))
</pre></div>
<p>
@@ -1736,7 +1742,7 @@ f6_grad_analytical <span style="color: #666666">=</span> <span style="color: #66
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10</span>):
f6_grad_analytical <span style="color: #666666">+=</span> i<span style="color: #666666">*</span>x<span style="color: #666666">**</span>(i<span style="color: #666666">-1</span>)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f6 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_grad_analytical))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f6 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_grad_analytical))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1758,7 +1764,7 @@ f7_grad <span style="color: #666666">=</span> grad(f7)
n <span style="color: #666666">=</span> <span style="color: #666666">2.0</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad(n)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad(n)))
<span style="color: #408080; font-style: italic"># The function f7 is an implementation of the factorial of n.</span>
<span style="color: #408080; font-style: italic"># By using the product rule, one can find that the derivative is:</span>
@@ -1771,7 +1777,7 @@ f7_grad_analytical <span style="color: #666666">=</span> <span style="color: #66
tmp <span style="color: #666666">*=</span> (n <span style="color: #666666">-</span> k)
f7_grad_analytical <span style="color: #666666">+=</span> tmp
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad_analytical))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad_analytical))
</pre></div>
<p>
Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
@@ -1797,7 +1803,7 @@ f8_grad <span style="color: #666666">=</span> grad(f8)
x <span style="color: #666666">=</span> <span style="color: #666666">8.4</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f8 is:&quot;</span>,f8_grad(x))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The derivative of f8 is:&quot;</span>,f8_grad(x))
</pre></div>
<p>
Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
@@ -1819,7 +1825,7 @@ f9_grad <span style="color: #666666">=</span> grad(f9)
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">0.0</span>])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f9 is:&quot;</span>,f9_grad(x))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The derivative of f9 is:&quot;</span>,f9_grad(x))
</pre></div>
<p>
Here we are told that the 'dot' function does not belong to Autograd's
@@ -1839,7 +1845,7 @@ f9_alternative_grad <span style="color: #666666">=</span> grad(f9_alternative)
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">3.0</span>,<span style="color: #666666">0.0</span>])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The gradient of f9 is:&quot;</span>,f9_alternative_grad(x))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;The gradient of f9 is:&quot;</span>,f9_alternative_grad(x))
<span style="color: #408080; font-style: italic"># The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively</span>
<span style="color: #408080; font-style: italic"># w.r.t x is (b_1, b_2).</span>
@@ -2112,7 +2118,7 @@ $$
fig <span style="color: #666666">=</span> pt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>gca(projection<span style="color: #666666">=</span><span style="color: #BA2121">&quot;3d&quot;</span>)
xmesh, ymesh <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mgrid[<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>,<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>]
xmesh, ymesh <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mgrid[<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50</span>j,<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50</span>j]
fmesh <span style="color: #666666">=</span> f(np<span style="color: #666666">.</span>array([xmesh, ymesh]))
ax<span style="color: #666666">.</span>plot_surface(xmesh, ymesh, fmesh)
</pre></div>
@@ -2144,7 +2150,7 @@ Run it!
alpha_opt <span style="color: #666666">=</span> sopt<span style="color: #666666">.</span>golden(f1d)
next_guess <span style="color: #666666">=</span> x <span style="color: #666666">+</span> alpha_opt <span style="color: #666666">*</span> s
guesses<span style="color: #666666">.</span>append(next_guess)
<span style="color: #008000; font-weight: bold">print</span>(next_guess)
<span style="color: #008000">print</span>(next_guess)
</pre></div>
<p>
What happened?
doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week39/ipynb/week39.ipynb
+6 -2
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 16, 2020**\n",
"Date: **Sep 22, 2020**\n",
"\n",
"Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -22,8 +22,12 @@
"\n",
"* Friday: Stochastic Gradient descent with examples and automatic differeantion\n",
"\n",
"Reading suggestions for both days: [Aurelien Geron's chapter 4](https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf) and [Murphy sections 8.3 and 8.5](https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf) \n",
"\n",
"## Thursday September 24\n",
"\n",
"\"Overview Video, why do we care about gradient methods?\":\"\"\n",
"\n",
"## Optimization, the central part of any Machine Learning algortithm\n",
"\n",
"Almost every problem in machine learning and data science starts with\n",
@@ -3039,5 +3043,5 @@
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 4
}
doc/src/week39/week39.do.txt
-1
View File
@@ -657,7 +657,6 @@ plt.show()
!split
===== Friday September 25 =====
"Overview Video, why stochastic gradient methods?":""
!split
===== Stochastic Gradient Descent =====