lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test

Browse Source
This commit is contained in:
mhjensen
2020-10-22 05:37:44 +02:00
parent 408d467cb1
commit 3737bf2caf
34 changed files with 3122 additions and 3072 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week43/week43.do.txt
+31 -25
View File
@@ -7,11 +7,14 @@ DATE: today
* Thursday: Wrapping up Recurrent Neural Networks and solving differential equations.
* Friday: Principal Component Analysis and Dimensionality Reduction
Reading suggestions for both days: "Aurelien Geron's chapters 8
We will also study the usage of "Autograd":"https://www.youtube.com/watch?v=fRf4l5qaX1M&ab_channel=AlexSmola" in computing gradients for deep learning. For the documentation of Autograd and examples see the lectures slides from "week 40":"https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html" and the "Autograd doucmentation":"https://github.com/HIPS/autograd".
!split
===== Recurrent Neural Networks =====
"Overview video":"https://www.youtube.com/watch?v=SEnXr6v2ifU&ab_channel=AlexanderAmini".
See also lecture on Thursday October 22 and examples from "week 42":"https://compphysics.github.io/MachineLearning/doc/pub/week42/html/week42.html".
!split
===== Solving ODEs with Deep Learning =====
@@ -308,7 +311,7 @@ f(z) = \frac{1}{1 + \exp{(-z)}}
It is possible to use other activations functions for the hidden layer also.
The output $\bm{x}_i^{\text{hidden} }$from each $i$-th hidden neuron is:
The output $\bm{x}_i^{\text{hidden}}$ from each $i$-th hidden neuron is:
$$
\bm{x}_i^{\text{hidden} } = f\big( \bm{z}_{i}^{\text{hidden}} \big)
@@ -316,7 +319,11 @@ $$
The outputs $\bm{x}_i^{\text{hidden} } $ are then sent to the output layer.
The output layer consists of one neuron in this case, and combines the output from each of the neurons in the hidden layers. The output layer combines the results from the hidden layer using some weights $ w_i^{\text{output}}$ and biases $b_i^{\text{output}}$. In this case, it is assumes that the number of neurons in the output layer is one.
The output layer consists of one neuron in this case, and combines the
output from each of the neurons in the hidden layers. The output layer
combines the results from the hidden layer using some weights $w_i^{\text{output}}$
and biases $b_i^{\text{output}}$. In this case,
it is assumes that the number of neurons in the output layer is one.
!split
===== Final technicalities III =====
@@ -356,7 +363,7 @@ b_1^{\text{output}} & \bm{w}_1^{\text{output}}
In this case we seek a continuous range of values since we are approximating a function. This means that after computing $\bm{z}_{1}^{\text{output}}$ the neural network has finished its feed forward step, and $\bm{z}_{1}^{\text{output}}$ is the final output of the network.
!split
===== Backpropagation =====
===== Back propagation =====
The next step is to decide how the parameters should be changed such that they minimize the cost function.
@@ -389,7 +396,7 @@ for a number of iterations or until $ \big|\big| \bm{\omega}_{\text{new} } - \bm
The value of $\lambda$ decides how large steps the algorithm must take
in the direction of $ \nabla_{\bm{\omega}} C(\bm{x}, \bm{\omega})$.
The notation $\nabla_{\bm{\omega}}$ express the gradient with respect
to the elements in $\bm{\omega$.
to the elements in $\bm{\omega}$.
In our case, we have to minimize the cost function $C(\bm{x}, P)$ with
respect to the two sets of weights and biases, that is for the hidden
@@ -742,8 +749,8 @@ Also, at $t = 0$ the population has the size $g(0) = g_0$, where $g_0$ is some c
In this example, similar network as for the exponential decay using Autograd has been used to solve the equation. However, as the implementation might suffer from e.g numerical instability
and high execution time (this might be more apparent in the examples solving PDEs),
a network has been constructed using TensorFlow also.
For comparison, the forward Euler method has been implemented in order to see how the networks performs compared to a numerical scheme.
using a library like TensorFlow is recommended.
Here, we stay with a more simple approach and implement for comparison, the simple forward Euler method.
!split
===== Setting up the problem =====
@@ -955,7 +962,7 @@ if __name__ == '__main__':
!split
===== Using forward Euler to solve the ODE =====
A straight-forward way of solving an ODE numerically, is to use Euler's method.
A straightforward way of solving an ODE numerically, is to use Euler's method.
Euler's method uses Taylor series to approximate the value at a function $f$ at a step $\Delta x$ from $x$:
@@ -1069,12 +1076,10 @@ if __name__ == '__main__':
plt.show()
!ec
Running the program gives
!split
===== Example: Solving the one dimensional Poisson equation using Autograd and TensorFlow =====
===== Example: Solving the one dimensional Poisson equation =====
The Poisson equation for $g(x)$ in one dimension is
@@ -1365,12 +1370,15 @@ f(x_2) \\
f(x_{N_x - 3}) \\
f(x_{N_x - 2})
\end{pmatrix} \\
A\bm{g} &= \bm{f}
\bm{A}\bm{g} &= \bm{f},
\end{aligned}
!et
which makes it possible to solve for the vector $\bm{g}$.
!split
===== Setting up the code =====
We can then compare the result from this numerical scheme with the output from our network using Autograd:
!bc pycod
@@ -1567,18 +1575,15 @@ if __name__ == '__main__':
!ec
The program prints out:
!bc
The max absolute difference between the analytical solution and DNN Autograd: 0.000464088
The max absolute difference between the analytical solution and numerical scheme: 0.00266858
!ec
!split
===== Partial Differential Equations =====
A partial differential equation (PDE) has a solution here the function is defined by multiple variables.
The equation may involve all kinds of combinations of which variables the function is differentiated with respect to.
A partial differential equation (PDE) has a solution here the function
is defined by multiple variables. The equation may involve all kinds
of combinations of which variables the function is differentiated with
respect to.
In general, a partial differential equation for a function $g(x_1,\dots,x_N)$ with $N$ variables may be expressed as
@@ -1623,24 +1628,24 @@ minimize is
!bt
\begin{equation*}
c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{\partial g(x_1,\dots,x_N) }{\partial x_1}, \dots , \frac{\partial g(x_1,\dots,x_N) }{\partial x_N}, \frac{\partial g(x_1,\dots,x_N) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(x_1,\dots,x_N) }{\partial x_N^n} \right) \right)^2
C\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{\partial g(x_1,\dots,x_N) }{\partial x_1}, \dots , \frac{\partial g(x_1,\dots,x_N) }{\partial x_N}, \frac{\partial g(x_1,\dots,x_N) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(x_1,\dots,x_N) }{\partial x_N^n} \right) \right)^2
\end{equation*}
!et
!split
===== More details =====
If we let $\bm{x = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following:
If we let $\bm{x} = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following:
!bt
\begin{equation*}
c\left(\bm{x}, P\right) = f\left( \left( \bm{x}, \frac{\partial g(\bm{x) }{\partial x_1}, \dots , \frac{\partial g(\bm{x) }{\partial x_N}, \frac{\partial g(\bm{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\bm{x) }{\partial x_N^n} \right) \right)^2
\end{equation*}
\[
C\left(\bm{x}, P\right) = f\left( \left( \bm{x}, \frac{\partial g(\bm{x}) }{\partial x_1}, \dots , \frac{\partial g(\bm{x}) }{\partial x_N}, \frac{\partial g(\bm{x}) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\bm{x}) }{\partial x_N^n} \right) \right)^2
\]
!et
If we also have $M$ different sets of values for $x_1, \dots, x_N$, that is $\bm{x}_i = \big(x_1^{(i)}, \dots, x_N^{(i)}\big)$ for $i = 1,\dots,M$ being the rows in matrix $X$, the cost function can be generalized into
!bt
\begin{equation*}
c\left(X, P \right) = \sum_{i=1}^M f\left( \left( \bm{x}_i, \frac{\partial g(\bm{x}_i) }{\partial x_1}, \dots , \frac{\partial g(\bm{x}_i) }{\partial x_N}, \frac{\partial g(\bm{x}_i) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\bm{x}_i) }{\partial x_N^n} \right) \right)^2
C\left(X, P \right) = \sum_{i=1}^M f\left( \left( \bm{x}_i, \frac{\partial g(\bm{x}_i) }{\partial x_1}, \dots , \frac{\partial g(\bm{x}_i) }{\partial x_N}, \frac{\partial g(\bm{x}_i) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\bm{x}_i) }{\partial x_N^n} \right) \right)^2.
\end{equation*}
!et
@@ -2395,6 +2400,7 @@ o "Introduction to Partial Differential Equations by A. Tveito, R. Winther":"htt
!split
===== Friday, Principal Component Analysis =====
"Overview video":"https://www.youtube.com/watch?v=fkf4IBRSeEc&ab_channel=SteveBrunton"
!split
===== Basic ideas of the Principal Component Analysis (PCA) =====