lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on jupyter-book

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2022-10-07 08:32:32 +02:00
parent 2b1016a356
commit e64776f3e3
31 changed files with 1764 additions and 1747 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapteroptimization.html
+68 -63
View File
@@ -781,17 +781,15 @@ line as discussed below.</p>
ensures convexity of a function <span class="math notranslate nohighlight">\(f\)</span>. We write <span class="math notranslate nohighlight">\(D_f\)</span> to denote the
domain of <span class="math notranslate nohighlight">\(f\)</span>, i.e the subset of <span class="math notranslate nohighlight">\(R^n\)</span> where <span class="math notranslate nohighlight">\(f\)</span> is defined. For more
details and proofs we refer to: [S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press](<a class="reference external" href="http://stanford.edu/boyd/cvxbook/">http://stanford.edu/boyd/cvxbook/</a>, 2004).</p>
<p><strong>First order condition.</strong></p>
<p>Suppose <span class="math notranslate nohighlight">\(f\)</span> is differentiable (i.e <span class="math notranslate nohighlight">\(\nabla f(x)\)</span> is well defined for
<p><strong>First order condition</strong>: Suppose <span class="math notranslate nohighlight">\(f\)</span> is differentiable (i.e <span class="math notranslate nohighlight">\(\nabla f(x)\)</span> is well defined for
all <span class="math notranslate nohighlight">\(x\)</span> in the domain of <span class="math notranslate nohighlight">\(f\)</span>). Then <span class="math notranslate nohighlight">\(f\)</span> is convex if and only if <span class="math notranslate nohighlight">\(D_f\)</span>
is a convex set and $<span class="math notranslate nohighlight">\(f(y) \geq f(x) + \nabla f(x)^T (y-x) \)</span><span class="math notranslate nohighlight">\( holds
for all \)</span>x,y \in D_f<span class="math notranslate nohighlight">\(. This condition means that for a convex function
is a convex set and <span class="math notranslate nohighlight">\(f(y) \geq f(x) + \nabla f(x)^T (y-x)\)</span> holds
for all <span class="math notranslate nohighlight">\(x,y \in D_f\)</span>. This condition means that for a convex function
the first order Taylor expansion (right hand side above) at any point
is a global under estimator of the function. To convince yourself you can
make a drawing of \)</span>f(x) = x^2+1<span class="math notranslate nohighlight">\( and draw the tangent line to \)</span>f(x)$ and
make a drawing of <span class="math notranslate nohighlight">\(f(x) = x^2+1\)</span> and draw the tangent line to <span class="math notranslate nohighlight">\(f(x)\)</span> and
note that it is always below the graph.</p>
<p><strong>Second order condition.</strong></p>
<p>Assume that <span class="math notranslate nohighlight">\(f\)</span> is twice
<p><strong>Second order condition</strong>: Assume that <span class="math notranslate nohighlight">\(f\)</span> is twice
differentiable, i.e the Hessian matrix exists at each point in
<span class="math notranslate nohighlight">\(D_f\)</span>. Then <span class="math notranslate nohighlight">\(f\)</span> is convex if and only if <span class="math notranslate nohighlight">\(D_f\)</span> is a convex set and its
Hessian is positive semi-definite for all <span class="math notranslate nohighlight">\(x\in D_f\)</span>. For a
@@ -940,11 +938,11 @@ which equals</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_96694/483257001.py:18: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_9414/483257001.py:18: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
ax = fig.gca(projection=&quot;3d&quot;)
</pre></div>
</div>
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x11db14850&gt;
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x13b240850&gt;
</pre></div>
</div>
<img alt="_images/chapteroptimization_61_2.png" src="_images/chapteroptimization_61_2.png" />
@@ -1002,7 +1000,7 @@ which equals</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x11e09b370&gt;]
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x13b8c72e0&gt;]
</pre></div>
</div>
<img alt="_images/chapteroptimization_69_1.png" src="_images/chapteroptimization_69_1.png" />
@@ -1259,11 +1257,11 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.2831603 4.55553537]
[[3.91511388]
[3.13030182]]
[[3.91511388]
[3.13030182]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.25881631 4.66111673]
[[4.01840062]
[2.89545727]]
[[4.01840062]
[2.89545727]]
</pre></div>
</div>
<img alt="_images/chapteroptimization_123_1.png" src="_images/chapteroptimization_123_1.png" />
@@ -1292,9 +1290,9 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.1509778 ]
[2.92461411]]
[4.13288373] [2.92817032]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[3.79441434]
[3.07608141]]
[3.80994952] [3.12302855]
</pre></div>
</div>
</div>
@@ -1365,10 +1363,10 @@ C_{\text{ridge}}(\beta) = \frac{1}{n}||X\beta -\mathbf{y}||^2 + \lambda ||\beta|
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.0795449 ]
[2.86893619]]
[[4.04785727]
[2.89298533]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[3.78596961]
[3.12387751]]
[[3.71441535]
[3.17942122]]
</pre></div>
</div>
<img alt="_images/chapteroptimization_132_1.png" src="_images/chapteroptimization_132_1.png" />
@@ -1618,15 +1616,15 @@ function.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
[[4.41170104]
[2.6431453 ]]
Eigenvalues of Hessian Matrix:[0.31228042 4.55571665]
[[4.42130182]
[2.83757843]]
Eigenvalues of Hessian Matrix:[0.27660123 4.17938393]
theta from own gd
[[4.41170104]
[2.6431453 ]]
[[4.42130182]
[2.83757843]]
theta from own sdg
[[4.39272691]
[2.63430285]]
[[4.44198566]
[2.79512696]]
</pre></div>
</div>
<img alt="_images/chapteroptimization_148_1.png" src="_images/chapteroptimization_148_1.png" />
@@ -2221,27 +2219,52 @@ The analytical derivative of f7 at n = 2 is: 1
<p>Assigning a value to the variable being differentiated with respect to</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd">import autograd.numpy as np</span>
<span class="sd">from autograd import grad</span>
<span class="sd">def f8(x): # Assume x is an array</span>
<span class="sd"> x[2] = 3</span>
<span class="sd"> return x*2</span>
<span class="sd">f8_grad = grad(f8)</span>
<span class="sd">x = 8.4</span>
<span class="sd">print(&quot;The derivative of f8 is:&quot;,f8_grad(x))</span>
<span class="sd">&quot;&quot;&quot;</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&#39;\nimport autograd.numpy as np\nfrom autograd import grad\ndef f8(x): # Assume x is an array\n x[2] = 3\n return x*2\n\nf8_grad = grad(f8)\n\nx = 8.4\n\nprint(&quot;The derivative of f8 is:&quot;,f8_grad(x))\n&#39;
</pre></div>
</div>
</div>
</div>
<p>Here, Autograd tells us that an ‘ArrayBox’ does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
<span class="k">def</span> <span class="nf">f8</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> <span class="c1"># Assume x is an array</span>
<span class="n">x</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="mi">3</span>
<span class="k">return</span> <span class="n">x</span><span class="o">*</span><span class="mi">2</span>
<span class="k">def</span> <span class="nf">f9</span><span class="p">(</span><span class="n">a</span><span class="p">):</span> <span class="c1"># Assume a is an array with 2 elements</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">2.0</span><span class="p">])</span>
<span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="n">f8_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f8</span><span class="p">)</span>
<span class="n">f9_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f9</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="mf">8.4</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">0.0</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The derivative of f8 is:&quot;</span><span class="p">,</span><span class="n">f8_grad</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The derivative of f9 is:&quot;</span><span class="p">,</span><span class="n">f9_grad</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">TypeError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="nn">Input In [22],</span> in <span class="ni">&lt;cell line: 11&gt;</span><span class="nt">()</span>
<span class="g g-Whitespace"> </span><span class="mi">7</span> <span class="n">f8_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f8</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">9</span> <span class="n">x</span> <span class="o">=</span> <span class="mf">8.4</span>
<span class="ne">---&gt; </span><span class="mi">11</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The derivative of f8 is:&quot;</span><span class="p">,</span><span class="n">f8_grad</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
<span class="ne">AttributeError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="nn">Input In [23],</span> in <span class="ni">&lt;cell line: 11&gt;</span><span class="nt">()</span>
<span class="g g-Whitespace"> </span><span class="mi">7</span> <span class="n">f9_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f9</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">9</span> <span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">0.0</span><span class="p">])</span>
<span class="ne">---&gt; </span><span class="mi">11</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The derivative of f9 is:&quot;</span><span class="p">,</span><span class="n">f9_grad</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/wrap_util.py:20,</span> in <span class="ni">unary_to_nary.&lt;locals&gt;.nary_operator.&lt;locals&gt;.nary_f</span><span class="nt">(*args, **kwargs)</span>
<span class="g g-Whitespace"> </span><span class="mi">18</span> <span class="k">else</span><span class="p">:</span>
@@ -2280,30 +2303,12 @@ The analytical derivative of f7 at n = 2 is: 1
<span class="g g-Whitespace"> </span><span class="mi">14</span> <span class="n">subargs</span> <span class="o">=</span> <span class="n">subvals</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="nb">zip</span><span class="p">(</span><span class="n">argnum</span><span class="p">,</span> <span class="n">x</span><span class="p">))</span>
<span class="ne">---&gt; </span><span class="mi">15</span> <span class="k">return</span> <span class="n">fun</span><span class="p">(</span><span class="o">*</span><span class="n">subargs</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="nn">Input In [22],</span> in <span class="ni">f8</span><span class="nt">(x)</span>
<span class="g g-Whitespace"> </span><span class="mi">3</span> <span class="k">def</span> <span class="nf">f8</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> <span class="c1"># Assume x is an array</span>
<span class="ne">----&gt; </span><span class="mi">4</span> <span class="n">x</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="mi">3</span>
<span class="g g-Whitespace"> </span><span class="mi">5</span> <span class="k">return</span> <span class="n">x</span><span class="o">*</span><span class="mi">2</span>
<span class="nn">Input In [23],</span> in <span class="ni">f9</span><span class="nt">(a)</span>
<span class="g g-Whitespace"> </span><span class="mi">3</span> <span class="k">def</span> <span class="nf">f9</span><span class="p">(</span><span class="n">a</span><span class="p">):</span> <span class="c1"># Assume a is an array with 2 elements</span>
<span class="g g-Whitespace"> </span><span class="mi">4</span> <span class="n">b</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">2.0</span><span class="p">])</span>
<span class="ne">----&gt; </span><span class="mi">5</span> <span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="ne">TypeError</span>: &#39;ArrayBox&#39; object does not support item assignment
</pre></div>
</div>
</div>
</div>
<p>Here, Autograd tells us that an ‘ArrayBox’ does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
<span class="k">def</span> <span class="nf">f9</span><span class="p">(</span><span class="n">a</span><span class="p">):</span> <span class="c1"># Assume a is an array with 2 elements</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">2.0</span><span class="p">])</span>
<span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="n">f9_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f9</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">0.0</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The derivative of f9 is:&quot;</span><span class="p">,</span><span class="n">f9_grad</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
<span class="ne">AttributeError</span>: &#39;ArrayBox&#39; object has no attribute &#39;dot&#39;
</pre></div>
</div>
</div>