lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

buidl

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-12-08 06:59:22 +01:00
parent d760cdfa85
commit 7ca589a654
94 changed files with 23333 additions and 2210 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter11.html
+319 -135
View File
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>14. Solving Differential Equations with Deep Learning &#8212; Applied Data Analysis and Machine Learning</title>
<title>15. Solving Differential Equations with Deep Learning &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet" />
<link href="_static/css/index.c5995385ac14fb8791e8eb36b4908be2.css" rel="stylesheet" />
@@ -53,7 +53,8 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="prev" title="13. Building a Feed Forward Neural Network" href="chapter10.html" />
<link rel="next" title="16. Convolutional Neural Networks" href="chapter12.html" />
<link rel="prev" title="14. Building a Feed Forward Neural Network" href="chapter10.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
@@ -198,6 +199,11 @@
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="clustering.html">
12. Clustering and Unsupervised Learning
</a>
</li>
</ul>
<p class="caption" role="heading">
<span class="caption-text">
@@ -207,17 +213,27 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
12. Neural networks
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
13. Building a Feed Forward Neural Network
14. Building a Feed Forward Neural Network
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
14. Solving Differential Equations with Deep Learning
15. Solving Differential Equations with Deep Learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter12.html">
16. Convolutional Neural Networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter13.html">
17. Recurrent neural networks: Overarching view
</a>
</li>
</ul>
@@ -293,90 +309,90 @@
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#example-exponential-decay">
14.1. Example: Exponential decay
15.1. Example: Exponential decay
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#reformulating-the-problem">
14.2. Reformulating the problem
15.2. Reformulating the problem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#gradient-descent">
14.3. Gradient descent
15.3. Gradient descent
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-code-for-solving-the-ode">
14.4. The code for solving the ODE
15.4. The code for solving the ODE
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer">
14.5. The network with one input layer, specified number of hidden layers, and one output layer
15.5. The network with one input layer, specified number of hidden layers, and one output layer
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#example-population-growth">
14.5.1. Example: Population growth
15.5.1. Example: Population growth
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-forward-euler-to-solve-the-ode">
14.6. Using forward Euler to solve the ODE
15.6. Using forward Euler to solve the ODE
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#solving-the-one-dimensional-poisson-equation">
14.7. Solving the one dimensional Poisson equation
15.7. Solving the one dimensional Poisson equation
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#comparing-with-a-numerical-scheme">
14.7.1. Comparing with a numerical scheme
15.7.1. Comparing with a numerical scheme
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#partial-differential-equations">
14.8. Partial Differential Equations
15.8. Partial Differential Equations
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#type-of-problem">
14.8.1. Type of problem
15.8.1. Type of problem
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#network-requirements">
14.8.2. Network requirements
15.8.2. Network requirements
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#example-the-diffusion-equation">
14.9. Example: The diffusion equation
15.9. Example: The diffusion equation
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#setting-up-the-network-using-autograd-the-full-program">
14.9.1. Setting up the network using Autograd; The full program
15.9.1. Setting up the network using Autograd; The full program
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#solving-the-wave-equation-with-neural-networks">
14.10. Solving the wave equation with Neural Networks
15.10. Solving the wave equation with Neural Networks
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#resources-on-differential-equations-and-deep-learning">
14.11. Resources on differential equations and deep learning
15.11. Resources on differential equations and deep learning
</a>
</li>
</ul>
@@ -390,8 +406,9 @@
<div>
<div class="tex2jax_ignore mathjax_ignore section" id="solving-differential-equations-with-deep-learning">
<h1><span class="section-number">14. </span>Solving Differential Equations with Deep Learning<a class="headerlink" href="#solving-differential-equations-with-deep-learning" title="Permalink to this headline"></a></h1>
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html chapter11.do.txt --><div class="tex2jax_ignore mathjax_ignore section" id="solving-differential-equations-with-deep-learning">
<h1><span class="section-number">15. </span>Solving Differential Equations with Deep Learning<a class="headerlink" href="#solving-differential-equations-with-deep-learning" title="Permalink to this headline"></a></h1>
<p>The Universal Approximation Theorem states that a neural network can
approximate any function at a single hidden layer along with one input
and output layer to any given precision.</p>
@@ -457,7 +474,7 @@ It might happen so that finding an analytical expression of the gradient of <spa
<p>Luckily, there exists libraries that makes the job for us through automatic differentiation.
Automatic differentiation is a method of finding the derivatives numerically with very high precision.</p>
<div class="section" id="example-exponential-decay">
<h2><span class="section-number">14.1. </span>Example: Exponential decay<a class="headerlink" href="#example-exponential-decay" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.1. </span>Example: Exponential decay<a class="headerlink" href="#example-exponential-decay" title="Permalink to this headline"></a></h2>
<p>An exponential decay of a quantity <span class="math notranslate nohighlight">\(g(x)\)</span> is described by the equation</p>
<!-- Equation labels as ordinary links -->
<div id="solve_expdec"></div>
@@ -512,7 +529,7 @@ g_t(x, P) = g_0 + x \cdot N(x, P)
\]</div>
</div>
<div class="section" id="reformulating-the-problem">
<h2><span class="section-number">14.2. </span>Reformulating the problem<a class="headerlink" href="#reformulating-the-problem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.2. </span>Reformulating the problem<a class="headerlink" href="#reformulating-the-problem" title="Permalink to this headline"></a></h2>
<p>We wish that our neural network manages to minimize a given cost function.</p>
<p>A reformulation of out equation, (<a class="reference external" href="#solveode">6</a>), must therefore be done,
such that it describes the problem a neural network can solve for.</p>
@@ -657,7 +674,7 @@ C(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i
<p>Here, gradient descent with a constant step size has been chosen.</p>
</div>
<div class="section" id="gradient-descent">
<h2><span class="section-number">14.3. </span>Gradient descent<a class="headerlink" href="#gradient-descent" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.3. </span>Gradient descent<a class="headerlink" href="#gradient-descent" title="Permalink to this headline"></a></h2>
<p>The idea of the gradient descent algorithm is to update parameters in
a direction where the cost function decreases goes to a minimum.</p>
<p>In general, the update of some parameters <span class="math notranslate nohighlight">\(\boldsymbol{\omega}\)</span> given a cost
@@ -686,7 +703,7 @@ P_{\text{output},\text{new}} &amp;= P_{\text{output}} - \lambda \nabla_{P_{\text
\end{split}\]</div>
</div>
<div class="section" id="the-code-for-solving-the-ode">
<h2><span class="section-number">14.4. </span>The code for solving the ODE<a class="headerlink" href="#the-code-for-solving-the-ode" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.4. </span>The code for solving the ODE<a class="headerlink" href="#the-code-for-solving-the-ode" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
@@ -846,12 +863,12 @@ P_{\text{output},\text{new}} &amp;= P_{\text{output}} - \lambda \nabla_{P_{\text
Max absolute difference: 0.0437499
</pre></div>
</div>
<img alt="_images/chapter11_47_2.png" src="_images/chapter11_47_2.png" />
<img alt="_images/chapter11_50_2.png" src="_images/chapter11_50_2.png" />
</div>
</div>
</div>
<div class="section" id="the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer">
<h2><span class="section-number">14.5. </span>The network with one input layer, specified number of hidden layers, and one output layer<a class="headerlink" href="#the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.5. </span>The network with one input layer, specified number of hidden layers, and one output layer<a class="headerlink" href="#the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer" title="Permalink to this headline"></a></h2>
<p>It is also possible to extend the construction of our network into a more general one, allowing the network to contain more than one hidden layers.</p>
<p>The number of neurons within each hidden layer are given as a list of integers in the program below.</p>
<div class="cell docutils container">
@@ -1025,106 +1042,14 @@ Max absolute difference: 0.0437499
return array(a, dtype, copy=False, order=order)
</pre></div>
</div>
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span>---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42376/2971492148.py in &lt;module&gt;
144 lmb = 0.001
145
--&gt; 146 P = solve_ode_deep_neural_network(x, num_hidden_neurons, num_iter, lmb)
147
148 res = g_trial_deep(x,P)
/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42376/2971492148.py in solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb)
119 # The cost_grad consist now of N_hidden + 1 arrays; the gradient w.r.t the weights and biases
120 # in the hidden layers and output layers evaluated at x.
--&gt; 121 cost_deep_grad = cost_function_deep_grad(P, x)
122
123 for l in range(N_hidden+1):
~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py in nary_f(*args, **kwargs)
18 else:
19 x = tuple(args[i] for i in argnum)
---&gt; 20 return unary_operator(unary_f, x, *nary_op_args, **nary_op_kwargs)
21 return nary_f
22 return nary_operator
~/anaconda3/lib/python3.8/site-packages/autograd/differential_operators.py in grad(fun, x)
23 arguments as `fun`, but returns the gradient instead. The function `fun`
24 should be scalar-valued. The gradient has the same type as the argument.&quot;&quot;&quot;
---&gt; 25 vjp, ans = _make_vjp(fun, x)
26 if not vspace(ans).size == 1:
27 raise TypeError(&quot;Grad only applies to real scalar-output functions. &quot;
~/anaconda3/lib/python3.8/site-packages/autograd/core.py in make_vjp(fun, x)
8 def make_vjp(fun, x):
9 start_node = VJPNode.new_root()
---&gt; 10 end_value, end_node = trace(start_node, fun, x)
11 if end_node is None:
12 def vjp(g): return vspace(x).zeros()
~/anaconda3/lib/python3.8/site-packages/autograd/tracer.py in trace(start_node, fun, x)
8 with trace_stack.new_trace() as t:
9 start_box = new_box(x, t, start_node)
---&gt; 10 end_box = fun(start_box)
11 if isbox(end_box) and end_box._trace == start_box._trace:
12 return end_box._value, end_box._node
~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py in unary_f(x)
13 else:
14 subargs = subvals(args, zip(argnum, x))
---&gt; 15 return fun(*subargs, **kwargs)
16 if isinstance(argnum, int):
17 x = args[argnum]
/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42376/2971492148.py in cost_function_deep(P, x)
67
68 # Evaluate the trial function with the current parameters P
---&gt; 69 g_t = g_trial_deep(x,P)
70
71 # Find the derivative w.r.t x of the neural network
/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42376/2971492148.py in g_trial_deep(x, params, g0)
57 # The trial solution using the deep neural network:
58 def g_trial_deep(x,params, g0 = 10):
---&gt; 59 return g0 + x*deep_neural_network(params, x)
60
61 # The right side of the ODE:
/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42376/2971492148.py in deep_neural_network(deep_params, x)
37
38 z_hidden = np.matmul(w_hidden, x_prev)
---&gt; 39 x_hidden = sigmoid(z_hidden)
40
41 # Update x_prev such that next layer can use the output from this layer
/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42376/2971492148.py in sigmoid(z)
5
6 def sigmoid(z):
----&gt; 7 return 1/(1 + np.exp(-z))
8
9 # The neural network with one input layer and one output layer,
~/anaconda3/lib/python3.8/site-packages/autograd/tracer.py in f_wrapped(*args, **kwargs)
40 if f_wrapped in notrace_primitives[node_constructor]:
41 return f_wrapped(*argvals, **kwargs)
---&gt; 42 parents = tuple(box._node for _ , box in boxed_args)
43 argnums = tuple(argnum for argnum, _ in boxed_args)
44 ans = f_wrapped(*argvals, **kwargs)
~/anaconda3/lib/python3.8/site-packages/autograd/tracer.py in &lt;genexpr&gt;(.0)
40 if f_wrapped in notrace_primitives[node_constructor]:
41 return f_wrapped(*argvals, **kwargs)
---&gt; 42 parents = tuple(box._node for _ , box in boxed_args)
43 argnums = tuple(argnum for argnum, _ in boxed_args)
44 ans = f_wrapped(*argvals, **kwargs)
KeyboardInterrupt:
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Final cost: 0.119936
</pre></div>
</div>
<img alt="_images/chapter11_52_3.png" src="_images/chapter11_52_3.png" />
</div>
</div>
<div class="section" id="example-population-growth">
<h3><span class="section-number">14.5.1. </span>Example: Population growth<a class="headerlink" href="#example-population-growth" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">15.5.1. </span>Example: Population growth<a class="headerlink" href="#example-population-growth" title="Permalink to this headline"></a></h3>
<p>A logistic model of population growth assumes that a population converges toward an equilibrium.
The population growth can be modeled by</p>
<!-- Equation labels as ordinary links -->
@@ -1335,11 +1260,25 @@ g(t) = \frac{Ag_0}{g_0 + (A - g_0)\exp(-\alpha A t)}
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Initial cost: 0.221805
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/MortenImac/anaconda3/lib/python3.8/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify &#39;dtype=object&#39; when creating the ndarray
return array(a, dtype, copy=False, order=order)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Final cost: 0.000417932
The max absolute difference between the solutions is: 0.00424909
</pre></div>
</div>
<img alt="_images/chapter11_58_3.png" src="_images/chapter11_58_3.png" />
</div>
</div>
</div>
</div>
<div class="section" id="using-forward-euler-to-solve-the-ode">
<h2><span class="section-number">14.6. </span>Using forward Euler to solve the ODE<a class="headerlink" href="#using-forward-euler-to-solve-the-ode" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.6. </span>Using forward Euler to solve the ODE<a class="headerlink" href="#using-forward-euler-to-solve-the-ode" title="Permalink to this headline"></a></h2>
<p>A straightforward way of solving an ODE numerically, is to use Euler’s method.</p>
<p>Euler’s method uses Taylor series to approximate the value at a function <span class="math notranslate nohighlight">\(f\)</span> at a step <span class="math notranslate nohighlight">\(\Delta x\)</span> from <span class="math notranslate nohighlight">\(x\)</span>:</p>
<div class="math notranslate nohighlight">
@@ -1452,10 +1391,27 @@ extending the program that uses the network using Autograd:</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Initial cost: 0.221805
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/MortenImac/anaconda3/lib/python3.8/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify &#39;dtype=object&#39; when creating the ndarray
return array(a, dtype, copy=False, order=order)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Final cost: 0.000417932
The max absolute difference between the solutions is: 0.00424909
Max absolute difference between Euler method and analytical: 0.011225
Max absolute difference between deep neural network and analytical: 0.00424909
</pre></div>
</div>
<img alt="_images/chapter11_66_3.png" src="_images/chapter11_66_3.png" />
<img alt="_images/chapter11_66_4.png" src="_images/chapter11_66_4.png" />
</div>
</div>
</div>
<div class="section" id="solving-the-one-dimensional-poisson-equation">
<h2><span class="section-number">14.7. </span>Solving the one dimensional Poisson equation<a class="headerlink" href="#solving-the-one-dimensional-poisson-equation" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.7. </span>Solving the one dimensional Poisson equation<a class="headerlink" href="#solving-the-one-dimensional-poisson-equation" title="Permalink to this headline"></a></h2>
<p>The Poisson equation for <span class="math notranslate nohighlight">\(g(x)\)</span> in one dimension is</p>
<!-- Equation labels as ordinary links -->
<div id="poisson"></div>
@@ -1657,9 +1613,23 @@ g(x) = x(1 - x)\exp(x)
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/MortenImac/anaconda3/lib/python3.8/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify &#39;dtype=object&#39; when creating the ndarray
return array(a, dtype, copy=False, order=order)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Initial cost: 457.256
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Final cost: 0.00310113
The max absolute difference between the solutions is: 0.000464088
</pre></div>
</div>
<img alt="_images/chapter11_79_3.png" src="_images/chapter11_79_3.png" />
</div>
</div>
<div class="section" id="comparing-with-a-numerical-scheme">
<h3><span class="section-number">14.7.1. </span>Comparing with a numerical scheme<a class="headerlink" href="#comparing-with-a-numerical-scheme" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">15.7.1. </span>Comparing with a numerical scheme<a class="headerlink" href="#comparing-with-a-numerical-scheme" title="Permalink to this headline"></a></h3>
<p>The Poisson equation is possible to solve using Taylor series to approximate the second derivative.</p>
<p>Using Taylor series, the second derivative can be expressed as</p>
<div class="math notranslate nohighlight">
@@ -1933,11 +1903,27 @@ f(x_{N_x - 2})
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Initial cost: 457.256
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/MortenImac/anaconda3/lib/python3.8/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify &#39;dtype=object&#39; when creating the ndarray
return array(a, dtype, copy=False, order=order)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Final cost: 0.00310113
The max absolute difference between the analytical solution and DNN Autograd: 0.000464088
The max absolute difference between the analytical solution and numerical scheme: 0.00266858
</pre></div>
</div>
<img alt="_images/chapter11_91_3.png" src="_images/chapter11_91_3.png" />
<img alt="_images/chapter11_91_4.png" src="_images/chapter11_91_4.png" />
</div>
</div>
</div>
</div>
<div class="section" id="partial-differential-equations">
<h2><span class="section-number">14.8. </span>Partial Differential Equations<a class="headerlink" href="#partial-differential-equations" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.8. </span>Partial Differential Equations<a class="headerlink" href="#partial-differential-equations" title="Permalink to this headline"></a></h2>
<p>A partial differential equation (PDE) has a solution here the function
is defined by multiple variables. The equation may involve all kinds
of combinations of which variables the function is differentiated with
@@ -1953,7 +1939,7 @@ respect to.</p>
\]</div>
<p>where <span class="math notranslate nohighlight">\(f\)</span> is an expression involving all kinds of possible mixed derivatives of <span class="math notranslate nohighlight">\(g(x_1,\dots,x_N)\)</span> up to an order <span class="math notranslate nohighlight">\(n\)</span>. In order for the solution to be unique, some additional conditions must also be given.</p>
<div class="section" id="type-of-problem">
<h3><span class="section-number">14.8.1. </span>Type of problem<a class="headerlink" href="#type-of-problem" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">15.8.1. </span>Type of problem<a class="headerlink" href="#type-of-problem" title="Permalink to this headline"></a></h3>
<p>The problem our network must solve for, is similar to the ODE case.
We must have a trial solution <span class="math notranslate nohighlight">\(g_t\)</span> at hand.</p>
<p>For instance, the trial solution could be expressed as</p>
@@ -1968,7 +1954,7 @@ The neural network <span class="math notranslate nohighlight">\(N(x_1,\dots,x_N,
<p>The role of the function <span class="math notranslate nohighlight">\(h_2(x_1,\dots,x_N,N(x_1,\dots,x_N,P))\)</span>, is to ensure that the output of <span class="math notranslate nohighlight">\(N(x_1,\dots,x_N,P)\)</span> is zero when <span class="math notranslate nohighlight">\(g_t(x_1,\dots,x_N)\)</span> is evaluated at the values of <span class="math notranslate nohighlight">\(x_1,\dots,x_N\)</span> where the given conditions must be satisfied. The function <span class="math notranslate nohighlight">\(h_1(x_1,\dots,x_N)\)</span> should alone make <span class="math notranslate nohighlight">\(g_t(x_1,\dots,x_N)\)</span> satisfy the conditions.</p>
</div>
<div class="section" id="network-requirements">
<h3><span class="section-number">14.8.2. </span>Network requirements<a class="headerlink" href="#network-requirements" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">15.8.2. </span>Network requirements<a class="headerlink" href="#network-requirements" title="Permalink to this headline"></a></h3>
<p>The network tries then the minimize the cost function following the
same ideas as described for the ODE case, but now with more than one
variables to consider. The concept still remains the same; find a set
@@ -1994,7 +1980,7 @@ C\left(X, P \right) = \sum_{i=1}^M f\left( \left( \boldsymbol{x}_i, \frac{\parti
</div>
</div>
<div class="section" id="example-the-diffusion-equation">
<h2><span class="section-number">14.9. </span>Example: The diffusion equation<a class="headerlink" href="#example-the-diffusion-equation" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.9. </span>Example: The diffusion equation<a class="headerlink" href="#example-the-diffusion-equation" title="Permalink to this headline"></a></h2>
<p>In one spatial dimension, the equation reads</p>
<div class="math notranslate nohighlight">
\[
@@ -2162,7 +2148,7 @@ mixed derivatives of <span class="math notranslate nohighlight">\(g(x,t)\)</span
</div>
</div>
<div class="section" id="setting-up-the-network-using-autograd-the-full-program">
<h3><span class="section-number">14.9.1. </span>Setting up the network using Autograd; The full program<a class="headerlink" href="#setting-up-the-network-using-autograd-the-full-program" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">15.9.1. </span>Setting up the network using Autograd; The full program<a class="headerlink" href="#setting-up-the-network-using-autograd-the-full-program" title="Permalink to this headline"></a></h3>
<p>Having set up the network, along with the trial solution and cost function, we can now see how the deep neural network performs by comparing the results to the analytical solution.</p>
<p>The analytical solution of our problem is</p>
<div class="math notranslate nohighlight">
@@ -2402,11 +2388,200 @@ Using TensorFlow results in a much better execution time. Try it!</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/MortenImac/anaconda3/lib/python3.8/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify &#39;dtype=object&#39; when creating the ndarray
return array(a, dtype, copy=False, order=order)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Initial cost: 41.05505310046362
</pre></div>
</div>
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span>---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47448/73752910.py in &lt;module&gt;
141 lmb = 0.01
142
--&gt; 143 P = solve_pde_deep_neural_network(x,t, num_hidden_neurons, num_iter, lmb)
144
145 ## Store the results
/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47448/73752910.py in solve_pde_deep_neural_network(x, t, num_neurons, num_iter, lmb)
118 # Let the update be done num_iter times
119 for i in range(num_iter):
--&gt; 120 cost_grad = cost_function_grad(P, x , t)
121
122 for l in range(N_hidden+1):
~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py in nary_f(*args, **kwargs)
18 else:
19 x = tuple(args[i] for i in argnum)
---&gt; 20 return unary_operator(unary_f, x, *nary_op_args, **nary_op_kwargs)
21 return nary_f
22 return nary_operator
~/anaconda3/lib/python3.8/site-packages/autograd/differential_operators.py in grad(fun, x)
23 arguments as `fun`, but returns the gradient instead. The function `fun`
24 should be scalar-valued. The gradient has the same type as the argument.&quot;&quot;&quot;
---&gt; 25 vjp, ans = _make_vjp(fun, x)
26 if not vspace(ans).size == 1:
27 raise TypeError(&quot;Grad only applies to real scalar-output functions. &quot;
~/anaconda3/lib/python3.8/site-packages/autograd/core.py in make_vjp(fun, x)
8 def make_vjp(fun, x):
9 start_node = VJPNode.new_root()
---&gt; 10 end_value, end_node = trace(start_node, fun, x)
11 if end_node is None:
12 def vjp(g): return vspace(x).zeros()
~/anaconda3/lib/python3.8/site-packages/autograd/tracer.py in trace(start_node, fun, x)
8 with trace_stack.new_trace() as t:
9 start_box = new_box(x, t, start_node)
---&gt; 10 end_box = fun(start_box)
11 if isbox(end_box) and end_box._trace == start_box._trace:
12 return end_box._value, end_box._node
~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py in unary_f(x)
13 else:
14 subargs = subvals(args, zip(argnum, x))
---&gt; 15 return fun(*subargs, **kwargs)
16 if isinstance(argnum, int):
17 x = args[argnum]
/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47448/73752910.py in cost_function(P, x, t)
78 g_t = g_trial(point,P)
79 g_t_jacobian = g_t_jacobian_func(point,P)
---&gt; 80 g_t_hessian = g_t_hessian_func(point,P)
81
82 g_t_dt = g_t_jacobian[1]
~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py in nary_f(*args, **kwargs)
18 else:
19 x = tuple(args[i] for i in argnum)
---&gt; 20 return unary_operator(unary_f, x, *nary_op_args, **nary_op_kwargs)
21 return nary_f
22 return nary_operator
~/anaconda3/lib/python3.8/site-packages/autograd/differential_operators.py in hessian(fun, x)
76 def hessian(fun, x):
77 &quot;Returns a function that computes the exact Hessian.&quot;
---&gt; 78 return jacobian(jacobian(fun))(x)
79
80 @unary_to_nary
~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py in nary_f(*args, **kwargs)
18 else:
19 x = tuple(args[i] for i in argnum)
---&gt; 20 return unary_operator(unary_f, x, *nary_op_args, **nary_op_kwargs)
21 return nary_f
22 return nary_operator
~/anaconda3/lib/python3.8/site-packages/autograd/differential_operators.py in jacobian(fun, x)
59 jacobian_shape = ans_vspace.shape + vspace(x).shape
60 grads = map(vjp, ans_vspace.standard_basis())
---&gt; 61 return np.reshape(np.stack(grads), jacobian_shape)
62
63 @unary_to_nary
~/anaconda3/lib/python3.8/site-packages/autograd/numpy/numpy_wrapper.py in stack(arrays, axis)
86 # primitives defined in this file
87
---&gt; 88 arrays = [array(arr) for arr in arrays]
89 if not arrays:
90 raise ValueError(&#39;need at least one array to stack&#39;)
~/anaconda3/lib/python3.8/site-packages/autograd/numpy/numpy_wrapper.py in &lt;listcomp&gt;(.0)
86 # primitives defined in this file
87
---&gt; 88 arrays = [array(arr) for arr in arrays]
89 if not arrays:
90 raise ValueError(&#39;need at least one array to stack&#39;)
~/anaconda3/lib/python3.8/site-packages/autograd/core.py in vjp(g)
12 def vjp(g): return vspace(x).zeros()
13 else:
---&gt; 14 def vjp(g): return backward_pass(g, end_node)
15 return vjp, end_value
16
~/anaconda3/lib/python3.8/site-packages/autograd/core.py in backward_pass(g, end_node)
19 for node in toposort(end_node):
20 outgrad = outgrads.pop(node)
---&gt; 21 ingrads = node.vjp(outgrad[0])
22 for parent, ingrad in zip(node.parents, ingrads):
23 outgrads[parent] = add_outgrads(outgrads.get(parent), ingrad)
~/anaconda3/lib/python3.8/site-packages/autograd/core.py in &lt;lambda&gt;(g)
65 &quot;VJP of {} wrt argnum 0 not defined&quot;.format(fun.__name__))
66 vjp = vjpfun(ans, *args, **kwargs)
---&gt; 67 return lambda g: (vjp(g),)
68 elif L == 2:
69 argnum_0, argnum_1 = argnums
~/anaconda3/lib/python3.8/site-packages/autograd/numpy/numpy_vjps.py in &lt;lambda&gt;(g)
421 A_ndim = anp.ndim(A)
422 B_meta = anp.metadata(B)
--&gt; 423 return lambda g: matmul_adjoint_1(A, g, A_ndim, B_meta)
424
425 defvjp(anp.matmul, matmul_vjp_0, matmul_vjp_1)
~/anaconda3/lib/python3.8/site-packages/autograd/numpy/numpy_vjps.py in matmul_adjoint_1(A, G, A_ndim, B_meta)
408 else: # We need to swap the last two axes of A
409 A = anp.swapaxes(A, A_ndim - 2, A_ndim - 1)
--&gt; 410 result = anp.matmul(A, G)
411 if B_is_vec:
412 result = anp.squeeze(result, anp.ndim(G) - 1)
~/anaconda3/lib/python3.8/site-packages/autograd/tracer.py in f_wrapped(*args, **kwargs)
43 argnums = tuple(argnum for argnum, _ in boxed_args)
44 ans = f_wrapped(*argvals, **kwargs)
---&gt; 45 node = node_constructor(ans, f_wrapped, argvals, kwargs, argnums, parents)
46 return new_box(ans, trace, node)
47 else:
~/anaconda3/lib/python3.8/site-packages/autograd/core.py in __init__(self, value, fun, args, kwargs, parent_argnums, parents)
34 raise NotImplementedError(&quot;VJP of {} wrt argnums {} not defined&quot;
35 .format(fun_name, parent_argnums))
---&gt; 36 self.vjp = vjpmaker(parent_argnums, value, args, kwargs)
37
38 def initialize_root(self):
~/anaconda3/lib/python3.8/site-packages/autograd/core.py in vjp_argnums(argnums, ans, args, kwargs)
75 &quot;VJP of {} wrt argnums 0, 1 not defined&quot;.format(fun.__name__))
76 vjp_0 = vjp_0_fun(ans, *args, **kwargs)
---&gt; 77 vjp_1 = vjp_1_fun(ans, *args, **kwargs)
78 return lambda g: (vjp_0(g), vjp_1(g))
79 else:
~/anaconda3/lib/python3.8/site-packages/autograd/numpy/numpy_vjps.py in matmul_vjp_1(ans, A, B)
420 def matmul_vjp_1(ans, A, B):
421 A_ndim = anp.ndim(A)
--&gt; 422 B_meta = anp.metadata(B)
423 return lambda g: matmul_adjoint_1(A, g, A_ndim, B_meta)
424
~/anaconda3/lib/python3.8/site-packages/autograd/tracer.py in f_wrapped(*args, **kwargs)
59 def f_wrapped(*args, **kwargs):
60 argvals = map(getval, args)
---&gt; 61 return f_raw(*argvals, **kwargs)
62 f_wrapped._is_primitive = True
63 return f_wrapped
~/anaconda3/lib/python3.8/site-packages/autograd/numpy/numpy_wrapper.py in metadata(A)
146 @notrace_primitive
147 def metadata(A):
--&gt; 148 return _np.shape(A), _np.ndim(A), _np.result_type(A), _np.iscomplexobj(A)
149
150 @notrace_primitive
KeyboardInterrupt:
</pre></div>
</div>
</div>
</div>
</div>
</div>
<div class="section" id="solving-the-wave-equation-with-neural-networks">
<h2><span class="section-number">14.10. </span>Solving the wave equation with Neural Networks<a class="headerlink" href="#solving-the-wave-equation-with-neural-networks" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.10. </span>Solving the wave equation with Neural Networks<a class="headerlink" href="#solving-the-wave-equation-with-neural-networks" title="Permalink to this headline"></a></h2>
<p>The wave equation is</p>
<div class="math notranslate nohighlight">
\[
@@ -2693,7 +2868,7 @@ g(x,t) = \sin(\pi x)\cos(\pi t) - \sin(\pi x)\sin(\pi t)
</div>
</div>
<div class="section" id="resources-on-differential-equations-and-deep-learning">
<h2><span class="section-number">14.11. </span>Resources on differential equations and deep learning<a class="headerlink" href="#resources-on-differential-equations-and-deep-learning" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">15.11. </span>Resources on differential equations and deep learning<a class="headerlink" href="#resources-on-differential-equations-and-deep-learning" title="Permalink to this headline"></a></h2>
<ol class="simple">
<li><p><a class="reference external" href="https://pdfs.semanticscholar.org/d061/df393e0e8fbfd0ea24976458b7d42419040d.pdf">Artificial neural networks for solving ordinary and partial differential equations by I.E. Lagaris et al</a></p></li>
<li><p><a class="reference external" href="https://becominghuman.ai/neural-networks-for-solving-differential-equations-fa230ac5e04c">Neural networks for solving differential equations by A. Honchar</a></p></li>
@@ -2737,10 +2912,19 @@ g(x,t) = \sin(\pi x)\cos(\pi t) - \sin(\pi x)\sin(\pi t)
<i class="prevnext-label fas fa-angle-left"></i>
<div class="prevnext-info">
<p class="prevnext-label">previous</p>
<p class="prevnext-title"><span class="section-number">13. </span>Building a Feed Forward Neural Network</p>
<p class="prevnext-title"><span class="section-number">14. </span>Building a Feed Forward Neural Network</p>
</div>
</a>
</div>
<div id="next">
<a class="right-next" href="chapter12.html" title="next page">
<div class="prevnext-info">
<p class="prevnext-label">next</p>
<p class="prevnext-title"><span class="section-number">16. </span>Convolutional Neural Networks</p>
</div>
<i class="prevnext-label fas fa-angle-right"></i>
</a>
</div>
</div>