diff --git a/doc/pub/week43/html/._week43-bs000.html b/doc/pub/week43/html/._week43-bs000.html index 5b34a0ba8..67ebb8e8e 100644 --- a/doc/pub/week43/html/._week43-bs000.html +++ b/doc/pub/week43/html/._week43-bs000.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({-
diff --git a/doc/pub/week43/html/._week43-bs001.html b/doc/pub/week43/html/._week43-bs001.html index 4714db0d4..18029d21e 100644 --- a/doc/pub/week43/html/._week43-bs001.html +++ b/doc/pub/week43/html/._week43-bs001.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Overview video. diff --git a/doc/pub/week43/html/._week43-bs003.html b/doc/pub/week43/html/._week43-bs003.html index 9c2adb8d4..32b2bc0dd 100644 --- a/doc/pub/week43/html/._week43-bs003.html +++ b/doc/pub/week43/html/._week43-bs003.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The Universal Approximation Theorem states that a neural network can diff --git a/doc/pub/week43/html/._week43-bs004.html b/doc/pub/week43/html/._week43-bs004.html index 39cf34075..0efbb055e 100644 --- a/doc/pub/week43/html/._week43-bs004.html +++ b/doc/pub/week43/html/._week43-bs004.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({An ordinary differential equation (ODE) is an equation involving functions having one variable. diff --git a/doc/pub/week43/html/._week43-bs005.html b/doc/pub/week43/html/._week43-bs005.html index 61d6bd835..c8a60d28c 100644 --- a/doc/pub/week43/html/._week43-bs005.html +++ b/doc/pub/week43/html/._week43-bs005.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Let the trial solution \( g_t(x) \) be diff --git a/doc/pub/week43/html/._week43-bs006.html b/doc/pub/week43/html/._week43-bs006.html index 82f9aa18e..c9d00a0b0 100644 --- a/doc/pub/week43/html/._week43-bs006.html +++ b/doc/pub/week43/html/._week43-bs006.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({For the minimization to be defined, we need to have a cost function at hand to minimize. diff --git a/doc/pub/week43/html/._week43-bs007.html b/doc/pub/week43/html/._week43-bs007.html index f867887a5..3487488fd 100644 --- a/doc/pub/week43/html/._week43-bs007.html +++ b/doc/pub/week43/html/._week43-bs007.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({To perform the minimization using gradient descent, the gradient of \( C\left(\boldsymbol{x}, P\right) \) is needed. diff --git a/doc/pub/week43/html/._week43-bs008.html b/doc/pub/week43/html/._week43-bs008.html index 53beb5a0b..b17a842a1 100644 --- a/doc/pub/week43/html/._week43-bs008.html +++ b/doc/pub/week43/html/._week43-bs008.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({An exponential decay of a quantity \( g(x) \) is described by the equation diff --git a/doc/pub/week43/html/._week43-bs009.html b/doc/pub/week43/html/._week43-bs009.html index d731e390f..93ca9cbdb 100644 --- a/doc/pub/week43/html/._week43-bs009.html +++ b/doc/pub/week43/html/._week43-bs009.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The program will use a neural network to solve diff --git a/doc/pub/week43/html/._week43-bs010.html b/doc/pub/week43/html/._week43-bs010.html index 2438c3260..817974cd1 100644 --- a/doc/pub/week43/html/._week43-bs010.html +++ b/doc/pub/week43/html/._week43-bs010.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({In this network, there are no weights and bias at the input layer, so \( P = \{ P_{\text{hidden}}, P_{\text{output}} \} \). diff --git a/doc/pub/week43/html/._week43-bs012.html b/doc/pub/week43/html/._week43-bs012.html index 1ad0f8a89..cbd958a4d 100644 --- a/doc/pub/week43/html/._week43-bs012.html +++ b/doc/pub/week43/html/._week43-bs012.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({We wish that our neural network manages to minimize a given cost function. diff --git a/doc/pub/week43/html/._week43-bs013.html b/doc/pub/week43/html/._week43-bs013.html index 3a49a07c6..820c8344c 100644 --- a/doc/pub/week43/html/._week43-bs013.html +++ b/doc/pub/week43/html/._week43-bs013.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The left hand side and right hand side of (8) must be computed separately, and then the neural network must choose weights and biases, contained in \( P \), such that the sides are equal as best as possible. diff --git a/doc/pub/week43/html/._week43-bs014.html b/doc/pub/week43/html/._week43-bs014.html index 0da56dce0..50b88fe00 100644 --- a/doc/pub/week43/html/._week43-bs014.html +++ b/doc/pub/week43/html/._week43-bs014.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({If the neural network evaluates \( g_t(x, P) \) at more values for \( x \), say \( N \) values \( x_i \) for \( i = 1, \dots, N \), then the total error to minimize becomes diff --git a/doc/pub/week43/html/._week43-bs015.html b/doc/pub/week43/html/._week43-bs015.html index 46cc77e10..2f1707d77 100644 --- a/doc/pub/week43/html/._week43-bs015.html +++ b/doc/pub/week43/html/._week43-bs015.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9). diff --git a/doc/pub/week43/html/._week43-bs016.html b/doc/pub/week43/html/._week43-bs016.html index 885aa95bb..4d6435e03 100644 --- a/doc/pub/week43/html/._week43-bs016.html +++ b/doc/pub/week43/html/._week43-bs016.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \) and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron at the input layer is: diff --git a/doc/pub/week43/html/._week43-bs017.html b/doc/pub/week43/html/._week43-bs017.html index 0dd5c4161..9981c6ffc 100644 --- a/doc/pub/week43/html/._week43-bs017.html +++ b/doc/pub/week43/html/._week43-bs017.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The result after weighting the inputs at the \( i \)-th hidden neuron can be written as a vector: diff --git a/doc/pub/week43/html/._week43-bs018.html b/doc/pub/week43/html/._week43-bs018.html index f68d4ab27..ba83a99b8 100644 --- a/doc/pub/week43/html/._week43-bs018.html +++ b/doc/pub/week43/html/._week43-bs018.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The vector \( \boldsymbol{p}_{i, \text{hidden}}^T \) constitutes each row in \( P_{\text{hidden} } \), which contains the weights for the neural network to minimize according to (9). diff --git a/doc/pub/week43/html/._week43-bs019.html b/doc/pub/week43/html/._week43-bs019.html index 5c42f6b3e..c68d54609 100644 --- a/doc/pub/week43/html/._week43-bs019.html +++ b/doc/pub/week43/html/._week43-bs019.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The procedure of weighting the output neuron \( j \) in the hidden layer to the \( i \)-th neuron in the output layer is similar as for the hidden layer described previously. diff --git a/doc/pub/week43/html/._week43-bs020.html b/doc/pub/week43/html/._week43-bs020.html index 3fd6c690c..fa092a70e 100644 --- a/doc/pub/week43/html/._week43-bs020.html +++ b/doc/pub/week43/html/._week43-bs020.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following way of weighting the inputs from the hidden layer: diff --git a/doc/pub/week43/html/._week43-bs021.html b/doc/pub/week43/html/._week43-bs021.html index 6536855d8..a53d66669 100644 --- a/doc/pub/week43/html/._week43-bs021.html +++ b/doc/pub/week43/html/._week43-bs021.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The next step is to decide how the parameters should be changed such that they minimize the cost function. diff --git a/doc/pub/week43/html/._week43-bs022.html b/doc/pub/week43/html/._week43-bs022.html index 9192d9756..de6cdbb4b 100644 --- a/doc/pub/week43/html/._week43-bs022.html +++ b/doc/pub/week43/html/._week43-bs022.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The idea of the gradient descent algorithm is to update parameters in diff --git a/doc/pub/week43/html/._week43-bs023.html b/doc/pub/week43/html/._week43-bs023.html index 0bad80e07..84a302189 100644 --- a/doc/pub/week43/html/._week43-bs023.html +++ b/doc/pub/week43/html/._week43-bs023.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({diff --git a/doc/pub/week43/html/._week43-bs024.html b/doc/pub/week43/html/._week43-bs024.html index a20407e5d..a783fddf0 100644 --- a/doc/pub/week43/html/._week43-bs024.html +++ b/doc/pub/week43/html/._week43-bs024.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({It is also possible to extend the construction of our network into a more general one, allowing the network to contain more than one hidden layers. diff --git a/doc/pub/week43/html/._week43-bs025.html b/doc/pub/week43/html/._week43-bs025.html index 32c4027f0..ec93ce5e7 100644 --- a/doc/pub/week43/html/._week43-bs025.html +++ b/doc/pub/week43/html/._week43-bs025.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({A logistic model of population growth assumes that a population converges toward an equilibrium. diff --git a/doc/pub/week43/html/._week43-bs026.html b/doc/pub/week43/html/._week43-bs026.html index 5009c0dc5..a70407947 100644 --- a/doc/pub/week43/html/._week43-bs026.html +++ b/doc/pub/week43/html/._week43-bs026.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Here, we will model a population \( g(t) \) in an environment having carrying capacity \( A \). diff --git a/doc/pub/week43/html/._week43-bs027.html b/doc/pub/week43/html/._week43-bs027.html index fad0a913d..9e4abf48e 100644 --- a/doc/pub/week43/html/._week43-bs027.html +++ b/doc/pub/week43/html/._week43-bs027.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({We will get a slightly different trial solution, as the boundary conditions are different diff --git a/doc/pub/week43/html/._week43-bs028.html b/doc/pub/week43/html/._week43-bs028.html index fd7b7de13..f8c0afd00 100644 --- a/doc/pub/week43/html/._week43-bs028.html +++ b/doc/pub/week43/html/._week43-bs028.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The network will be the similar as for the exponential decay example, but with some small modifications for our problem. diff --git a/doc/pub/week43/html/._week43-bs029.html b/doc/pub/week43/html/._week43-bs029.html index 5fdcc43de..1b3903e99 100644 --- a/doc/pub/week43/html/._week43-bs029.html +++ b/doc/pub/week43/html/._week43-bs029.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({A straightforward way of solving an ODE numerically, is to use Euler's method. diff --git a/doc/pub/week43/html/._week43-bs030.html b/doc/pub/week43/html/._week43-bs030.html index 26e10ace3..82ebfda96 100644 --- a/doc/pub/week43/html/._week43-bs030.html +++ b/doc/pub/week43/html/._week43-bs030.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The Poisson equation for \( g(x) \) in one dimension is diff --git a/doc/pub/week43/html/._week43-bs031.html b/doc/pub/week43/html/._week43-bs031.html index 9cae8f8ab..7752a47e3 100644 --- a/doc/pub/week43/html/._week43-bs031.html +++ b/doc/pub/week43/html/._week43-bs031.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Here, the function \( g(x) \) to solve for follows the equation diff --git a/doc/pub/week43/html/._week43-bs032.html b/doc/pub/week43/html/._week43-bs032.html index e9f705002..1775691d3 100644 --- a/doc/pub/week43/html/._week43-bs032.html +++ b/doc/pub/week43/html/._week43-bs032.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({diff --git a/doc/pub/week43/html/._week43-bs033.html b/doc/pub/week43/html/._week43-bs033.html index 69ba54e02..5c5fb5133 100644 --- a/doc/pub/week43/html/._week43-bs033.html +++ b/doc/pub/week43/html/._week43-bs033.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The Poisson equation is possible to solve using Taylor series to approximate the second derivative. diff --git a/doc/pub/week43/html/._week43-bs034.html b/doc/pub/week43/html/._week43-bs034.html index a3fee85b7..f912c3c26 100644 --- a/doc/pub/week43/html/._week43-bs034.html +++ b/doc/pub/week43/html/._week43-bs034.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({We can then compare the result from this numerical scheme with the output from our network using Autograd: diff --git a/doc/pub/week43/html/._week43-bs035.html b/doc/pub/week43/html/._week43-bs035.html index 726ad43a0..767d63038 100644 --- a/doc/pub/week43/html/._week43-bs035.html +++ b/doc/pub/week43/html/._week43-bs035.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({A partial differential equation (PDE) has a solution here the function diff --git a/doc/pub/week43/html/._week43-bs036.html b/doc/pub/week43/html/._week43-bs036.html index 4da3dba65..2069800a9 100644 --- a/doc/pub/week43/html/._week43-bs036.html +++ b/doc/pub/week43/html/._week43-bs036.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The problem our network must solve for, is similar to the ODE case. diff --git a/doc/pub/week43/html/._week43-bs037.html b/doc/pub/week43/html/._week43-bs037.html index 43bf2f3ae..e89ab0d0f 100644 --- a/doc/pub/week43/html/._week43-bs037.html +++ b/doc/pub/week43/html/._week43-bs037.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The network tries then the minimize the cost function following the diff --git a/doc/pub/week43/html/._week43-bs038.html b/doc/pub/week43/html/._week43-bs038.html index c4552ab87..e995f6151 100644 --- a/doc/pub/week43/html/._week43-bs038.html +++ b/doc/pub/week43/html/._week43-bs038.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({If we let \( \boldsymbol{x} = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: diff --git a/doc/pub/week43/html/._week43-bs039.html b/doc/pub/week43/html/._week43-bs039.html index 1e3d4d217..46aa3df4b 100644 --- a/doc/pub/week43/html/._week43-bs039.html +++ b/doc/pub/week43/html/._week43-bs039.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({In one spatial dimension, the equation reads diff --git a/doc/pub/week43/html/._week43-bs040.html b/doc/pub/week43/html/._week43-bs040.html index 6a597ed03..96326c426 100644 --- a/doc/pub/week43/html/._week43-bs040.html +++ b/doc/pub/week43/html/._week43-bs040.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({For this case, we want to find \( g(x,t) \) such that diff --git a/doc/pub/week43/html/._week43-bs041.html b/doc/pub/week43/html/._week43-bs041.html index 29365503d..36219dfce 100644 --- a/doc/pub/week43/html/._week43-bs041.html +++ b/doc/pub/week43/html/._week43-bs041.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The only change to do here, is to extend our network such that diff --git a/doc/pub/week43/html/._week43-bs042.html b/doc/pub/week43/html/._week43-bs042.html index bb30c8c32..5e70b842b 100644 --- a/doc/pub/week43/html/._week43-bs042.html +++ b/doc/pub/week43/html/._week43-bs042.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The cost function must then iterate through the given arrays diff --git a/doc/pub/week43/html/._week43-bs043.html b/doc/pub/week43/html/._week43-bs043.html index d7f62e5c4..aeac8278b 100644 --- a/doc/pub/week43/html/._week43-bs043.html +++ b/doc/pub/week43/html/._week43-bs043.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The Jacobian is used because the program must find the derivative of diff --git a/doc/pub/week43/html/._week43-bs044.html b/doc/pub/week43/html/._week43-bs044.html index 29cae84ea..7d53260ed 100644 --- a/doc/pub/week43/html/._week43-bs044.html +++ b/doc/pub/week43/html/._week43-bs044.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Having set up the network, along with the trial solution and cost function, we can now see how the deep neural network performs by comparing the results to the analytical solution. diff --git a/doc/pub/week43/html/._week43-bs045.html b/doc/pub/week43/html/._week43-bs045.html index 43e6de5a1..db71ecc87 100644 --- a/doc/pub/week43/html/._week43-bs045.html +++ b/doc/pub/week43/html/._week43-bs045.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The wave equation is diff --git a/doc/pub/week43/html/._week43-bs046.html b/doc/pub/week43/html/._week43-bs046.html index 49c16ace5..0cd0a41bf 100644 --- a/doc/pub/week43/html/._week43-bs046.html +++ b/doc/pub/week43/html/._week43-bs046.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The wave equation to solve for, is diff --git a/doc/pub/week43/html/._week43-bs047.html b/doc/pub/week43/html/._week43-bs047.html index b854166ad..ba58aa617 100644 --- a/doc/pub/week43/html/._week43-bs047.html +++ b/doc/pub/week43/html/._week43-bs047.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The analytical solution for our specific problem, is diff --git a/doc/pub/week43/html/._week43-bs049.html b/doc/pub/week43/html/._week43-bs049.html index ccc1c8dc7..d563d365a 100644 --- a/doc/pub/week43/html/._week43-bs049.html +++ b/doc/pub/week43/html/._week43-bs049.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({diff --git a/doc/pub/week43/html/._week43-bs050.html b/doc/pub/week43/html/._week43-bs050.html index 98a20ffc8..2ff34c66e 100644 --- a/doc/pub/week43/html/._week43-bs050.html +++ b/doc/pub/week43/html/._week43-bs050.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Overview video diff --git a/doc/pub/week43/html/._week43-bs052.html b/doc/pub/week43/html/._week43-bs052.html index a985322ba..d06f0460e 100644 --- a/doc/pub/week43/html/._week43-bs052.html +++ b/doc/pub/week43/html/._week43-bs052.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The principal component analysis deals with the problem of fitting a diff --git a/doc/pub/week43/html/._week43-bs053.html b/doc/pub/week43/html/._week43-bs053.html index 04b559612..30f60d019 100644 --- a/doc/pub/week43/html/._week43-bs053.html +++ b/doc/pub/week43/html/._week43-bs053.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Before we discuss the PCA theorem, we need to remind ourselves about diff --git a/doc/pub/week43/html/._week43-bs054.html b/doc/pub/week43/html/._week43-bs054.html index cae1130d6..6953bf1a3 100644 --- a/doc/pub/week43/html/._week43-bs054.html +++ b/doc/pub/week43/html/._week43-bs054.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression diff --git a/doc/pub/week43/html/._week43-bs055.html b/doc/pub/week43/html/._week43-bs055.html index 5f4b1c09f..8c4c9906f 100644 --- a/doc/pub/week43/html/._week43-bs055.html +++ b/doc/pub/week43/html/._week43-bs055.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The Numpy function np.cov calculates the covariance elements using diff --git a/doc/pub/week43/html/._week43-bs056.html b/doc/pub/week43/html/._week43-bs056.html index 9ce214653..6c05a923b 100644 --- a/doc/pub/week43/html/._week43-bs056.html +++ b/doc/pub/week43/html/._week43-bs056.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({The previous example can be converted into the correlation matrix by diff --git a/doc/pub/week43/html/._week43-bs057.html b/doc/pub/week43/html/._week43-bs057.html index e9a9386d4..199e51dac 100644 --- a/doc/pub/week43/html/._week43-bs057.html +++ b/doc/pub/week43/html/._week43-bs057.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({We whow here how we can set up the correlation matrix using pandas, as done in this simple code diff --git a/doc/pub/week43/html/._week43-bs058.html b/doc/pub/week43/html/._week43-bs058.html index 86d386b42..2b219f27d 100644 --- a/doc/pub/week43/html/._week43-bs058.html +++ b/doc/pub/week43/html/._week43-bs058.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({diff --git a/doc/pub/week43/html/._week43-bs059.html b/doc/pub/week43/html/._week43-bs059.html index 77a753ec9..ed6efeea8 100644 --- a/doc/pub/week43/html/._week43-bs059.html +++ b/doc/pub/week43/html/._week43-bs059.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as diff --git a/doc/pub/week43/html/._week43-bs060.html b/doc/pub/week43/html/._week43-bs060.html index 61c87c892..6e15e3a04 100644 --- a/doc/pub/week43/html/._week43-bs060.html +++ b/doc/pub/week43/html/._week43-bs060.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as diff --git a/doc/pub/week43/html/._week43-bs061.html b/doc/pub/week43/html/._week43-bs061.html index 6b7d165f9..f7f785824 100644 --- a/doc/pub/week43/html/._week43-bs061.html +++ b/doc/pub/week43/html/._week43-bs061.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here. @@ -327,7 +329,7 @@ $$
We will use a simple example first with two-dimensional data -drawn from a multivariate normal distribution with the following mean and covariance matrix: +drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): $$ \mu = (-1,2) \qquad \Sigma = \begin{bmatrix} 4 & 2 \\ 2 & 2 @@ -316,8 +318,8 @@ $$ $$ Note that the mean refers to each column of data. -We will generate \( n = 1000 \) points \( X = \{ x_1, \ldots, x_N \} \) from -this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). +We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from +this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values.
The following Python code aids in setting up the data and writing out the design matrix. @@ -337,7 +339,7 @@ X = np.r
Now we are going to implement the PCA algorithm. We will break it down into various substeps. -
The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is @@ -352,7 +354,7 @@ $$ When you are done with these steps, print out \( \mu_n \) to verify it is close to \( \mu \) and plot your mean centered data to verify it is -centered at the origin! Compare your code with the functionality from Scikit-Learn discussed above. +centered at the origin! The following code elements perform these operations using pandas or using our own functionality for doing so. The latter, using numpy is rather simple through the mean() function.
@@ -374,7 +376,7 @@ variance. The diagonal covariance matrix elements will then be one, while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our specific case. -
Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation @@ -414,9 +416,9 @@ plt.show()
Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. -The plot shows how the data are clustered around a line with slope close to one. Is this expected? +The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed. -
Now we are ready to solve for the principal components! To do so we diff --git a/doc/pub/week43/html/._week43-bs063.html b/doc/pub/week43/html/._week43-bs063.html index 8bcc8f0d4..b53560b7d 100644 --- a/doc/pub/week43/html/._week43-bs063.html +++ b/doc/pub/week43/html/._week43-bs063.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({We assume now that we have a design matrix \( \boldsymbol{X} \) which has been @@ -313,15 +315,6 @@ overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \). -
-We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as -$$ -J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}}_i)^2, -$$ - -with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix -\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality. -
The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diff --git a/doc/pub/week43/html/._week43-bs064.html b/doc/pub/week43/html/._week43-bs064.html index 3b3488d30..1817841ba 100644 --- a/doc/pub/week43/html/._week43-bs064.html +++ b/doc/pub/week43/html/._week43-bs064.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({-To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as -$$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{n}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), -$$ - -which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as -$$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). -$$ - -Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that -$$ -z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i, -$$ - -where the vectors on the rhs are known. +To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
diff --git a/doc/pub/week43/html/._week43-bs065.html b/doc/pub/week43/html/._week43-bs065.html index f1fad3bc5..9c436aef5 100644 --- a/doc/pub/week43/html/._week43-bs065.html +++ b/doc/pub/week43/html/._week43-bs065.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({-We have now found the unknown parameters \( z_{i0} \). These correspond to the projected coordinates and we can write -$$ -J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2. -$$ - -
-We can show that the variance of the projected coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by -$$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2, -$$ - -since the expectation value of -$$ -\mathbb{E}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \mathbb{E}[z_{i0}]= \boldsymbol{w}_0^T\mathbb{E}[\boldsymbol{x}_i]=0, -$$ - -where we have used the fact that our data are centered. - -
-Recalling our definition of the covariance as -$$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T], -$$ - -we have thus that -$$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0. -$$ +
We are almost there, we have obtained a relation between minimizing diff --git a/doc/pub/week43/html/._week43-bs066.html b/doc/pub/week43/html/._week43-bs066.html index 9e07f1869..7a6563d95 100644 --- a/doc/pub/week43/html/._week43-bs066.html +++ b/doc/pub/week43/html/._week43-bs066.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({We could trivially maximize the variance of the projection (and diff --git a/doc/pub/week43/html/._week43-bs067.html b/doc/pub/week43/html/._week43-bs067.html index d496369c2..5476031d8 100644 --- a/doc/pub/week43/html/._week43-bs067.html +++ b/doc/pub/week43/html/._week43-bs067.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2. diff --git a/doc/pub/week43/html/._week43-bs068.html b/doc/pub/week43/html/._week43-bs068.html index acb68c7d3..1643e64a0 100644 --- a/doc/pub/week43/html/._week43-bs068.html +++ b/doc/pub/week43/html/._week43-bs068.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm. diff --git a/doc/pub/week43/html/._week43-bs069.html b/doc/pub/week43/html/._week43-bs069.html index 76f663a70..450940c24 100644 --- a/doc/pub/week43/html/._week43-bs069.html +++ b/doc/pub/week43/html/._week43-bs069.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The diff --git a/doc/pub/week43/html/._week43-bs070.html b/doc/pub/week43/html/._week43-bs070.html index 7b30fc5c6..900a91d69 100644 --- a/doc/pub/week43/html/._week43-bs070.html +++ b/doc/pub/week43/html/._week43-bs070.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({diff --git a/doc/pub/week43/html/._week43-bs071.html b/doc/pub/week43/html/._week43-bs071.html index 9f925d170..f8b29450e 100644 --- a/doc/pub/week43/html/._week43-bs071.html +++ b/doc/pub/week43/html/._week43-bs071.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to diff --git a/doc/pub/week43/html/._week43-bs072.html b/doc/pub/week43/html/._week43-bs072.html index 16bf363db..758953a69 100644 --- a/doc/pub/week43/html/._week43-bs072.html +++ b/doc/pub/week43/html/._week43-bs072.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({One problem with the preceding implementation of PCA is that it requires the whole training set to fit in diff --git a/doc/pub/week43/html/._week43-bs073.html b/doc/pub/week43/html/._week43-bs073.html index 60c2b4630..7b2c6f576 100644 --- a/doc/pub/week43/html/._week43-bs073.html +++ b/doc/pub/week43/html/._week43-bs073.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic diff --git a/doc/pub/week43/html/._week43-bs074.html b/doc/pub/week43/html/._week43-bs074.html index 0517159e2..ae16d5662 100644 --- a/doc/pub/week43/html/._week43-bs074.html +++ b/doc/pub/week43/html/._week43-bs074.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({diff --git a/doc/pub/week43/html/._week43-bs075.html b/doc/pub/week43/html/._week43-bs075.html index de8009f41..cdfbe4448 100644 --- a/doc/pub/week43/html/._week43-bs075.html +++ b/doc/pub/week43/html/._week43-bs075.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction diff --git a/doc/pub/week43/html/._week43-bs076.html b/doc/pub/week43/html/._week43-bs076.html index 020800275..0c39e5bfa 100644 --- a/doc/pub/week43/html/._week43-bs076.html +++ b/doc/pub/week43/html/._week43-bs076.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn. diff --git a/doc/pub/week43/html/week43-bs.html b/doc/pub/week43/html/week43-bs.html index 5b34a0ba8..67ebb8e8e 100644 --- a/doc/pub/week43/html/week43-bs.html +++ b/doc/pub/week43/html/week43-bs.html @@ -41,139 +41,140 @@ Automatically generated HTML file from DocOnce source
@@ -211,84 +212,85 @@ MathJax.Hub.Config({-
diff --git a/doc/pub/week43/html/week43-reveal.html b/doc/pub/week43/html/week43-reveal.html index 0b372e64c..8caa40010 100644 --- a/doc/pub/week43/html/week43-reveal.html +++ b/doc/pub/week43/html/week43-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
-
@@ -159,6 +159,7 @@ MathJax.Hub.Config({
Overview video.
@@ -186,7 +187,7 @@ See also lecture on Thursday October 22 and examples from Solving ODEs with Deep Learning
+
The Universal Approximation Theorem states that a neural network can
@@ -196,7 +197,7 @@ and output layer to any given precision.
An ordinary differential equation (ODE) is an equation involving functions having one variable.
@@ -225,7 +226,7 @@ for the solution to be unique.
Let the trial solution \( g_t(x) \) be
@@ -258,7 +259,7 @@ As described previously, an optimization method could be used to minimize the pa
For the minimization to be defined, we need to have a cost function at hand to minimize.
@@ -294,7 +295,7 @@ The neural net should then find the parameters \( P \) that minimizes the cost f
To perform the minimization using gradient descent, the gradient of \( C\left(\boldsymbol{x}, P\right) \) is needed.
@@ -307,7 +308,7 @@ Automatic differentiation is a method of finding the derivatives numerically wit
An exponential decay of a quantity \( g(x) \) is described by the equation
@@ -341,7 +342,7 @@ Having an analytical solution at hand, it is possible to use it to compare how w
The program will use a neural network to solve
@@ -363,7 +364,7 @@ In this example, \( \gamma = 2 \) and \( g_0 = 10 \).
In this network, there are no weights and bias at the input layer, so \( P = \{ P_{\text{hidden}}, P_{\text{output}} \} \).
@@ -405,7 +406,7 @@ $$
We wish that our neural network manages to minimize a given cost function.
@@ -443,7 +444,7 @@ is fulfilled as best as possible.
The left hand side and right hand side of (8) must be computed separately, and then the neural network must choose weights and biases, contained in \( P \), such that the sides are equal as best as possible.
@@ -477,7 +478,7 @@ for an input value \( x \).
If the neural network evaluates \( g_t(x, P) \) at more values for \( x \), say \( N \) values \( x_i \) for \( i = 1, \dots, N \), then the total error to minimize becomes
@@ -511,7 +512,7 @@ $$
For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9).
@@ -524,7 +525,7 @@ The input layer will consist of \( N_{\text{input} } \) neurons, passing its ele
For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \) and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron at the input layer is:
@@ -548,7 +549,7 @@ $$
The result after weighting the inputs at the \( i \)-th hidden neuron can be written as a vector:
@@ -573,7 +574,7 @@ $$
The vector \( \boldsymbol{p}_{i, \text{hidden}}^T \) constitutes each row in \( P_{\text{hidden} } \), which contains the weights for the neural network to minimize according to (9).
@@ -615,7 +616,7 @@ it is assumes that the number of neurons in the output layer is one.
The procedure of weighting the output neuron \( j \) in the hidden layer to the \( i \)-th neuron in the output layer is similar as for the hidden layer described previously.
@@ -638,7 +639,7 @@ $$
Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following way of weighting the inputs from the hidden layer:
@@ -662,7 +663,7 @@ In this case we seek a continuous range of values since we are approximating a f
The next step is to decide how the parameters should be changed such that they minimize the cost function.
@@ -685,7 +686,7 @@ Here, gradient descent with a constant step size has been chosen.
The idea of the gradient descent algorithm is to update parameters in
@@ -732,7 +733,7 @@ $$
@@ -886,7 +887,7 @@ $$
It is also possible to extend the construction of our network into a more general one, allowing the network to contain more than one hidden layers.
@@ -1060,7 +1061,7 @@ The number of neurons within each hidden layer are given as a list of integers i
A logistic model of population growth assumes that a population converges toward an equilibrium.
@@ -1087,7 +1088,7 @@ Here, we stay with a more simple approach and implement for comparison, the simp
Here, we will model a population \( g(t) \) in an environment having carrying capacity \( A \).
@@ -1110,7 +1111,7 @@ In this example, we let \( \alpha = 2 \), \( A = 1 \), and \( g_0 = 1.2 \).
We will get a slightly different trial solution, as the boundary conditions are different
@@ -1140,7 +1141,7 @@ $$
The network will be the similar as for the exponential decay example, but with some small modifications for our problem.
@@ -1316,7 +1317,7 @@ The network will be the similar as for the exponential decay example, but with s
A straightforward way of solving an ODE numerically, is to use Euler's method.
@@ -1454,7 +1455,7 @@ extending the program that uses the network using Autograd:
The Poisson equation for \( g(x) \) in one dimension is
@@ -1489,7 +1490,7 @@ In addition, it could be interesting to see how a typical method for numerically
Here, the function \( g(x) \) to solve for follows the equation
@@ -1535,7 +1536,7 @@ $$
@@ -1695,7 +1696,7 @@ $$
The Poisson equation is possible to solve using Taylor series to approximate the second derivative.
@@ -1805,7 +1806,7 @@ which makes it possible to solve for the vector \( \boldsymbol{g} \).
We can then compare the result from this numerical scheme with the output from our network using Autograd:
@@ -2008,7 +2009,7 @@ We can then compare the result from this numerical scheme with the output from o
A partial differential equation (PDE) has a solution here the function
@@ -2033,7 +2034,7 @@ where \( f \) is an expression involving all kinds of possible mixed derivatives
The problem our network must solve for, is similar to the ODE case.
@@ -2058,7 +2059,7 @@ The role of the function \( h_2(x_1,\dots,x_N,N(x_1,\dots,x_N,P)) \), is to ensu
The network tries then the minimize the cost function following the
@@ -2083,7 +2084,7 @@ $$
If we let \( \boldsymbol{x} = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
@@ -2106,7 +2107,7 @@ $$
In one spatial dimension, the equation reads
@@ -2135,7 +2136,7 @@ with \( u(x) \) being some given function.
For this case, we want to find \( g(x,t) \) such that
@@ -2171,7 +2172,7 @@ First, we will look into how Autograd could be used in a network tailored to sol
The only change to do here, is to extend our network such that
@@ -2234,7 +2235,7 @@ network at each possible pair \( (x,t) \), given an array for the desired
The cost function must then iterate through the given arrays
@@ -2267,7 +2268,7 @@ since \( (0) = u(1) = 0 \) and \( u(x) = \sin(\pi x) \).
The Jacobian is used because the program must find the derivative of
@@ -2335,7 +2336,7 @@ mixed derivatives of \( g(x,t) \).
Having set up the network, along with the trial solution and cost function, we can now see how the deep neural network performs by comparing the results to the analytical solution.
@@ -2589,7 +2590,7 @@ Using TensorFlow results in a much better execution time. Try it!
The wave equation is
@@ -2622,7 +2623,7 @@ where \( \frac{\partial g(x,t)}{\partial t} \Big |_{t = 0} \) means the derivati
The wave equation to solve for, is
@@ -2656,7 +2657,7 @@ In this example, let \( c = 1 \) and \( u(x) = \sin(\pi x) \) and \( v(x) = -\pi
The analytical solution for our specific problem, is
@@ -2698,7 +2699,7 @@ $$
@@ -2928,7 +2929,7 @@ $$
Overview video
@@ -2948,7 +2949,7 @@ $$
Plans for week 43
Recurrent Neural Networks
Solving ODEs with Deep Learning
Ordinary Differential Equations
+Ordinary Differential Equations
The trial solution
+The trial solution
Minimization process
+Minimization process
Minimizing the cost function using gradient descent and automatic differentiation
+Minimizing the cost function using gradient descent and automatic differentiation
Example: Exponential decay
+Example: Exponential decay
The function to solve for
+The function to solve for
The trial solution
+The trial solution
To begin with, a trial solution \( g_t(t) \) must be chosen. A general trial solution for ordinary differential equations could be
@@ -378,7 +379,7 @@ with \( h_1(x) \) ensuring that \( g_t(x) \) satisfies some conditions and \( h_
Setup of Network
+Setup of Network
Reformulating the problem
+Reformulating the problem
More technicalities
+More technicalities
More details
+More details
A possible implementation of a neural network
+A possible implementation of a neural network
Technicalities
+Technicalities
Final technicalities I
+Final technicalities I
Final technicalities II
+Final technicalities II
Final technicalities III
+Final technicalities III
Final technicalities IV
+Final technicalities IV
Back propagation
+Back propagation
Gradient descent
+Gradient descent
The code for solving the ODE
+The code for solving the ODE
The network with one input layer, specified number of hidden layers, and one output layer
+The network with one input layer, specified number of hidden layers, and one output layer
Example: Population growth
+Example: Population growth
Setting up the problem
+Setting up the problem
The trial solution
+The trial solution
The program using Autograd
+The program using Autograd
Using forward Euler to solve the ODE
+Using forward Euler to solve the ODE
Example: Solving the one dimensional Poisson equation
+Example: Solving the one dimensional Poisson equation
The specific equation to solve for
+The specific equation to solve for
Solving the equation using Autograd
+Solving the equation using Autograd
Comparing with a numerical scheme
+Comparing with a numerical scheme
Setting up the code
+Setting up the code
Partial Differential Equations
+Partial Differential Equations
Type of problem
+Type of problem
Network requirements
+Network requirements
More details
+More details
Example: The diffusion equation
+Example: The diffusion equation
Defining the problem
+Defining the problem
Setting up the network using Autograd
+Setting up the network using Autograd
Setting up the network using Autograd; The trial solution
+Setting up the network using Autograd; The trial solution
Why the jacobian?
+Why the jacobian?
Setting up the network using Autograd; The full program
+Setting up the network using Autograd; The full program
Example: Solving the wave equation with Neural Networks
+Example: Solving the wave equation with Neural Networks
The problem to solve for
+The problem to solve for
The trial solution
+The trial solution
Setting up the network is done in similar matter as for the example of solving the diffusion equation.
The only things we have to change, is the trial solution such that it satisfies the conditions from (20) and the cost function.
@@ -2684,7 +2685,7 @@ Note that this trial solution satisfies the conditions only if \( u(0) = v(0) =
The analytical solution
+The analytical solution
Solving the wave equation - the full program using Autograd
+Solving the wave equation - the full program using Autograd
Resources on differential equations and deep learning
+Resources on differential equations and deep learning
Friday, Principal Component Analysis
+Friday, Principal Component Analysis
Basic ideas of the Principal Component Analysis (PCA)
+Basic ideas of the Principal Component Analysis (PCA)
Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. -The plot shows how the data are clustered around a line with slope close to one. Is this expected? +The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed. -
Now we are ready to solve for the principal components! To do so we
@@ -3626,7 +3627,7 @@ This code does not contain all the above elements, but it shows how we can use <
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
@@ -3635,17 +3636,6 @@ overline symbol. The matrix is defined in terms of the various column
vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension
\( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
-
-We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
-
The PCA theorem states that minimizing the above reconstruction error
corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which
@@ -3658,76 +3648,15 @@ eigenvectors of the covariance(correlations matrix).
-To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
-
-We have now found the unknown parameters \( z_{i0} \). These correspond to the projected coordinates and we can write
-
-We can show that the variance of the projected coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by
-
-Recalling our definition of the covariance as
-
We are almost there, we have obtained a relation between minimizing
@@ -3738,7 +3667,7 @@ of the projected data.
We could trivially maximize the variance of the projection (and
@@ -3800,7 +3729,7 @@ For more details, see for example Geometric Interpretation and link with Singular Value Decomposition
+
For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.
@@ -3808,7 +3737,7 @@ For a detailed demonstration of the geometric interpretation, see Principal Component Analysis
+
Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
@@ -3864,7 +3793,7 @@ X2D = X_centered.dot(W2)
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -3896,7 +3825,7 @@ variance that lies along the axis of each principal component.
@@ -3937,7 +3866,7 @@ We see that our training data after the PCA decomposition has a performance simi
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -3968,7 +3897,7 @@ X_reduced = pca.fit_transform(X)
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -3980,7 +3909,7 @@ instances arrive).
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -3991,7 +3920,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
@@ -4017,7 +3946,7 @@ X_reduced = rbf_pca.fit_transform(X)
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -4029,7 +3958,7 @@ these local relationships are best preserved (more details shortly).
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
diff --git a/doc/pub/week43/html/week43-solarized.html b/doc/pub/week43/html/week43-solarized.html
index 898b21c21..094fa80c7 100644
--- a/doc/pub/week43/html/week43-solarized.html
+++ b/doc/pub/week43/html/week43-solarized.html
@@ -61,139 +61,140 @@ div { text-align: justify; text-justify: inter-word; }
-
Overview video.
@@ -265,7 +268,7 @@ See also lecture on Thursday October 22 and examples from Solving ODEs with Deep Learning
+
The Universal Approximation Theorem states that a neural network can
@@ -275,7 +278,7 @@ and output layer to any given precision.
An ordinary differential equation (ODE) is an equation involving functions having one variable.
@@ -302,7 +305,7 @@ for the solution to be unique.
Let the trial solution \( g_t(x) \) be
@@ -333,7 +336,7 @@ As described previously, an optimization method could be used to minimize the pa
For the minimization to be defined, we need to have a cost function at hand to minimize.
@@ -365,7 +368,7 @@ The neural net should then find the parameters \( P \) that minimizes the cost f
To perform the minimization using gradient descent, the gradient of \( C\left(\boldsymbol{x}, P\right) \) is needed.
@@ -378,7 +381,7 @@ Automatic differentiation is a method of finding the derivatives numerically wit
An exponential decay of a quantity \( g(x) \) is described by the equation
@@ -408,7 +411,7 @@ Having an analytical solution at hand, it is possible to use it to compare how w
The program will use a neural network to solve
@@ -428,7 +431,7 @@ In this example, \( \gamma = 2 \) and \( g_0 = 10 \).
In this network, there are no weights and bias at the input layer, so \( P = \{ P_{\text{hidden}}, P_{\text{output}} \} \).
@@ -466,7 +469,7 @@ $$
We wish that our neural network manages to minimize a given cost function.
@@ -500,7 +503,7 @@ is fulfilled as best as possible.
The left hand side and right hand side of \eqref{nnmin} must be computed separately, and then the neural network must choose weights and biases, contained in \( P \), such that the sides are equal as best as possible.
@@ -530,7 +533,7 @@ for an input value \( x \).
If the neural network evaluates \( g_t(x, P) \) at more values for \( x \), say \( N \) values \( x_i \) for \( i = 1, \dots, N \), then the total error to minimize becomes
@@ -558,7 +561,7 @@ $$
For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.
@@ -571,7 +574,7 @@ The input layer will consist of \( N_{\text{input} } \) neurons, passing its ele
For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \) and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron at the input layer is:
@@ -593,7 +596,7 @@ $$
The result after weighting the inputs at the \( i \)-th hidden neuron can be written as a vector:
@@ -616,7 +619,7 @@ $$
The vector \( \boldsymbol{p}_{i, \text{hidden}}^T \) constitutes each row in \( P_{\text{hidden} } \), which contains the weights for the neural network to minimize according to \eqref{min}.
@@ -654,7 +657,7 @@ it is assumes that the number of neurons in the output layer is one.
The procedure of weighting the output neuron \( j \) in the hidden layer to the \( i \)-th neuron in the output layer is similar as for the hidden layer described previously.
@@ -675,7 +678,7 @@ $$
Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following way of weighting the inputs from the hidden layer:
@@ -697,7 +700,7 @@ In this case we seek a continuous range of values since we are approximating a f
The next step is to decide how the parameters should be changed such that they minimize the cost function.
@@ -718,7 +721,7 @@ Here, gradient descent with a constant step size has been chosen.
The idea of the gradient descent algorithm is to update parameters in
@@ -761,7 +764,7 @@ $$
@@ -914,7 +917,7 @@ $$
It is also possible to extend the construction of our network into a more general one, allowing the network to contain more than one hidden layers.
@@ -1087,7 +1090,7 @@ The number of neurons within each hidden layer are given as a list of integers i
A logistic model of population growth assumes that a population converges toward an equilibrium.
@@ -1112,7 +1115,7 @@ Here, we stay with a more simple approach and implement for comparison, the simp
Here, we will model a population \( g(t) \) in an environment having carrying capacity \( A \).
@@ -1133,7 +1136,7 @@ In this example, we let \( \alpha = 2 \), \( A = 1 \), and \( g_0 = 1.2 \).
We will get a slightly different trial solution, as the boundary conditions are different
@@ -1159,7 +1162,7 @@ $$
The network will be the similar as for the exponential decay example, but with some small modifications for our problem.
@@ -1334,7 +1337,7 @@ The network will be the similar as for the exponential decay example, but with s
A straightforward way of solving an ODE numerically, is to use Euler's method.
@@ -1463,7 +1466,7 @@ extending the program that uses the network using Autograd:
The Poisson equation for \( g(x) \) in one dimension is
@@ -1494,7 +1497,7 @@ In addition, it could be interesting to see how a typical method for numerically
Here, the function \( g(x) \) to solve for follows the equation
@@ -1532,7 +1535,7 @@ $$
@@ -1691,7 +1694,7 @@ $$
The Poisson equation is possible to solve using Taylor series to approximate the second derivative.
@@ -1789,7 +1792,7 @@ which makes it possible to solve for the vector \( \boldsymbol{g} \).
We can then compare the result from this numerical scheme with the output from our network using Autograd:
@@ -1991,7 +1994,7 @@ We can then compare the result from this numerical scheme with the output from o
A partial differential equation (PDE) has a solution here the function
@@ -2014,7 +2017,7 @@ where \( f \) is an expression involving all kinds of possible mixed derivatives
The problem our network must solve for, is similar to the ODE case.
@@ -2037,7 +2040,7 @@ The role of the function \( h_2(x_1,\dots,x_N,N(x_1,\dots,x_N,P)) \), is to ensu
The network tries then the minimize the cost function following the
@@ -2060,7 +2063,7 @@ $$
If we let \( \boldsymbol{x} = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
@@ -2079,7 +2082,7 @@ $$
In one spatial dimension, the equation reads
@@ -2104,7 +2107,7 @@ with \( u(x) \) being some given function.
For this case, we want to find \( g(x,t) \) such that
@@ -2136,7 +2139,7 @@ First, we will look into how Autograd could be used in a network tailored to sol
The only change to do here, is to extend our network such that
@@ -2198,7 +2201,7 @@ network at each possible pair \( (x,t) \), given an array for the desired
The cost function must then iterate through the given arrays
@@ -2227,7 +2230,7 @@ since \( (0) = u(1) = 0 \) and \( u(x) = \sin(\pi x) \).
The Jacobian is used because the program must find the derivative of
@@ -2294,7 +2297,7 @@ mixed derivatives of \( g(x,t) \).
Having set up the network, along with the trial solution and cost function, we can now see how the deep neural network performs by comparing the results to the analytical solution.
@@ -2545,7 +2548,7 @@ Using TensorFlow results in a much better execution time. Try it!
The wave equation is
@@ -2574,7 +2577,7 @@ where \( \frac{\partial g(x,t)}{\partial t} \Big |_{t = 0} \) means the derivati
The wave equation to solve for, is
@@ -2604,7 +2607,7 @@ In this example, let \( c = 1 \) and \( u(x) = \sin(\pi x) \) and \( v(x) = -\pi
The analytical solution for our specific problem, is
@@ -2640,7 +2643,7 @@ $$
@@ -2869,7 +2872,7 @@ $$
Overview video
@@ -2888,7 +2891,7 @@ $$
Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
-The plot shows how the data are clustered around a line with slope close to one. Is this expected?
+The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.
-
Now we are ready to solve for the principal components! To do so we
@@ -3511,7 +3514,7 @@ This code does not contain all the above elements, but it shows how we can use <
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
@@ -3520,15 +3523,6 @@ overline symbol. The matrix is defined in terms of the various column
vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension
\( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
-
-We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
-$$
-J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}}_i)^2,
-$$
-
-with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
-\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
-
The PCA theorem states that minimizing the above reconstruction error
corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which
@@ -3541,60 +3535,15 @@ eigenvectors of the covariance(correlations matrix).
-To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
-$$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{n}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
-$$
-
-which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
-$$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
-$$
-
-Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that
-$$
-z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i,
-$$
-
-where the vectors on the rhs are known.
+To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
-We have now found the unknown parameters \( z_{i0} \). These correspond to the projected coordinates and we can write
-$$
-J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2.
-$$
-
-
-We can show that the variance of the projected coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by
-$$
-\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2,
-$$
-
-since the expectation value of
-$$
-\mathbb{E}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \mathbb{E}[z_{i0}]= \boldsymbol{w}_0^T\mathbb{E}[\boldsymbol{x}_i]=0,
-$$
-
-where we have used the fact that our data are centered.
-
-
-Recalling our definition of the covariance as
-$$
-\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T],
-$$
-
-we have thus that
-$$
-\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0.
-$$
+
We are almost there, we have obtained a relation between minimizing
@@ -3605,7 +3554,7 @@ of the projected data.
We could trivially maximize the variance of the projection (and
@@ -3659,7 +3608,7 @@ For more details, see for example Geometric Interpretation and link with Singular Value Decomposition
+
For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.
@@ -3667,7 +3616,7 @@ For a detailed demonstration of the geometric interpretation, see Principal Component Analysis
+
Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
@@ -3722,7 +3671,7 @@ X2D = X_centered.dot(W2)
-
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -3754,7 +3703,7 @@ variance that lies along the axis of each principal component.
@@ -3795,7 +3744,7 @@ We see that our training data after the PCA decomposition has a performance simi
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -3825,7 +3774,7 @@ X_reduced = pca.fit_transform(X)
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -3837,7 +3786,7 @@ instances arrive).
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -3848,7 +3797,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
@@ -3877,7 +3826,7 @@ X_reduced = rbf_pca.fit_transform(X)
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -3889,7 +3838,7 @@ these local relationships are best preserved (more details shortly).
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
diff --git a/doc/pub/week43/html/week43.html b/doc/pub/week43/html/week43.html
index 7091c4c39..4810cb2ca 100644
--- a/doc/pub/week43/html/week43.html
+++ b/doc/pub/week43/html/week43.html
@@ -66,139 +66,140 @@ div { text-align: justify; text-justify: inter-word; }
-
Overview video.
@@ -270,7 +273,7 @@ See also lecture on Thursday October 22 and examples from Solving ODEs with Deep Learning
+
The Universal Approximation Theorem states that a neural network can
@@ -280,7 +283,7 @@ and output layer to any given precision.
An ordinary differential equation (ODE) is an equation involving functions having one variable.
@@ -307,7 +310,7 @@ for the solution to be unique.
Let the trial solution \( g_t(x) \) be
@@ -338,7 +341,7 @@ As described previously, an optimization method could be used to minimize the pa
For the minimization to be defined, we need to have a cost function at hand to minimize.
@@ -370,7 +373,7 @@ The neural net should then find the parameters \( P \) that minimizes the cost f
To perform the minimization using gradient descent, the gradient of \( C\left(\boldsymbol{x}, P\right) \) is needed.
@@ -383,7 +386,7 @@ Automatic differentiation is a method of finding the derivatives numerically wit
An exponential decay of a quantity \( g(x) \) is described by the equation
@@ -413,7 +416,7 @@ Having an analytical solution at hand, it is possible to use it to compare how w
The program will use a neural network to solve
@@ -433,7 +436,7 @@ In this example, \( \gamma = 2 \) and \( g_0 = 10 \).
In this network, there are no weights and bias at the input layer, so \( P = \{ P_{\text{hidden}}, P_{\text{output}} \} \).
@@ -471,7 +474,7 @@ $$
We wish that our neural network manages to minimize a given cost function.
@@ -505,7 +508,7 @@ is fulfilled as best as possible.
The left hand side and right hand side of \eqref{nnmin} must be computed separately, and then the neural network must choose weights and biases, contained in \( P \), such that the sides are equal as best as possible.
@@ -535,7 +538,7 @@ for an input value \( x \).
If the neural network evaluates \( g_t(x, P) \) at more values for \( x \), say \( N \) values \( x_i \) for \( i = 1, \dots, N \), then the total error to minimize becomes
@@ -563,7 +566,7 @@ $$
For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.
@@ -576,7 +579,7 @@ The input layer will consist of \( N_{\text{input} } \) neurons, passing its ele
For the \( i \)-th in the hidden layer with weight \( w_i^{\text{hidden} } \) and bias \( b_i^{\text{hidden} } \), the weighting from the \( j \)-th neuron at the input layer is:
@@ -598,7 +601,7 @@ $$
The result after weighting the inputs at the \( i \)-th hidden neuron can be written as a vector:
@@ -621,7 +624,7 @@ $$
The vector \( \boldsymbol{p}_{i, \text{hidden}}^T \) constitutes each row in \( P_{\text{hidden} } \), which contains the weights for the neural network to minimize according to \eqref{min}.
@@ -659,7 +662,7 @@ it is assumes that the number of neurons in the output layer is one.
The procedure of weighting the output neuron \( j \) in the hidden layer to the \( i \)-th neuron in the output layer is similar as for the hidden layer described previously.
@@ -680,7 +683,7 @@ $$
Expressing \( z_{1,j}^{\text{output}} \) as a vector gives the following way of weighting the inputs from the hidden layer:
@@ -702,7 +705,7 @@ In this case we seek a continuous range of values since we are approximating a f
The next step is to decide how the parameters should be changed such that they minimize the cost function.
@@ -723,7 +726,7 @@ Here, gradient descent with a constant step size has been chosen.
The idea of the gradient descent algorithm is to update parameters in
@@ -766,7 +769,7 @@ $$
@@ -919,7 +922,7 @@ $$
It is also possible to extend the construction of our network into a more general one, allowing the network to contain more than one hidden layers.
@@ -1092,7 +1095,7 @@ The number of neurons within each hidden layer are given as a list of integers i
A logistic model of population growth assumes that a population converges toward an equilibrium.
@@ -1117,7 +1120,7 @@ Here, we stay with a more simple approach and implement for comparison, the simp
Here, we will model a population \( g(t) \) in an environment having carrying capacity \( A \).
@@ -1138,7 +1141,7 @@ In this example, we let \( \alpha = 2 \), \( A = 1 \), and \( g_0 = 1.2 \).
We will get a slightly different trial solution, as the boundary conditions are different
@@ -1164,7 +1167,7 @@ $$
The network will be the similar as for the exponential decay example, but with some small modifications for our problem.
@@ -1339,7 +1342,7 @@ The network will be the similar as for the exponential decay example, but with s
A straightforward way of solving an ODE numerically, is to use Euler's method.
@@ -1468,7 +1471,7 @@ extending the program that uses the network using Autograd:
The Poisson equation for \( g(x) \) in one dimension is
@@ -1499,7 +1502,7 @@ In addition, it could be interesting to see how a typical method for numerically
Here, the function \( g(x) \) to solve for follows the equation
@@ -1537,7 +1540,7 @@ $$
@@ -1696,7 +1699,7 @@ $$
The Poisson equation is possible to solve using Taylor series to approximate the second derivative.
@@ -1794,7 +1797,7 @@ which makes it possible to solve for the vector \( \boldsymbol{g} \).
We can then compare the result from this numerical scheme with the output from our network using Autograd:
@@ -1996,7 +1999,7 @@ We can then compare the result from this numerical scheme with the output from o
A partial differential equation (PDE) has a solution here the function
@@ -2019,7 +2022,7 @@ where \( f \) is an expression involving all kinds of possible mixed derivatives
The problem our network must solve for, is similar to the ODE case.
@@ -2042,7 +2045,7 @@ The role of the function \( h_2(x_1,\dots,x_N,N(x_1,\dots,x_N,P)) \), is to ensu
The network tries then the minimize the cost function following the
@@ -2065,7 +2068,7 @@ $$
If we let \( \boldsymbol{x} = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
@@ -2084,7 +2087,7 @@ $$
In one spatial dimension, the equation reads
@@ -2109,7 +2112,7 @@ with \( u(x) \) being some given function.
For this case, we want to find \( g(x,t) \) such that
@@ -2141,7 +2144,7 @@ First, we will look into how Autograd could be used in a network tailored to sol
The only change to do here, is to extend our network such that
@@ -2203,7 +2206,7 @@ network at each possible pair \( (x,t) \), given an array for the desired
The cost function must then iterate through the given arrays
@@ -2232,7 +2235,7 @@ since \( (0) = u(1) = 0 \) and \( u(x) = \sin(\pi x) \).
The Jacobian is used because the program must find the derivative of
@@ -2299,7 +2302,7 @@ mixed derivatives of \( g(x,t) \).
Having set up the network, along with the trial solution and cost function, we can now see how the deep neural network performs by comparing the results to the analytical solution.
@@ -2550,7 +2553,7 @@ Using TensorFlow results in a much better execution time. Try it!
The wave equation is
@@ -2579,7 +2582,7 @@ where \( \frac{\partial g(x,t)}{\partial t} \Big |_{t = 0} \) means the derivati
The wave equation to solve for, is
@@ -2609,7 +2612,7 @@ In this example, let \( c = 1 \) and \( u(x) = \sin(\pi x) \) and \( v(x) = -\pi
The analytical solution for our specific problem, is
@@ -2645,7 +2648,7 @@ $$
@@ -2874,7 +2877,7 @@ $$
Overview video
@@ -2893,7 +2896,7 @@ $$
Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
-The plot shows how the data are clustered around a line with slope close to one. Is this expected?
+The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.
-
Now we are ready to solve for the principal components! To do so we
@@ -3516,7 +3519,7 @@ This code does not contain all the above elements, but it shows how we can use <
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
@@ -3525,15 +3528,6 @@ overline symbol. The matrix is defined in terms of the various column
vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension
\( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
-
-We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
-$$
-J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}}_i)^2,
-$$
-
-with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
-\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
-
The PCA theorem states that minimizing the above reconstruction error
corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which
@@ -3546,60 +3540,15 @@ eigenvectors of the covariance(correlations matrix).
-To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
-$$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{n}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
-$$
-
-which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
-$$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
-$$
-
-Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that
-$$
-z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i,
-$$
-
-where the vectors on the rhs are known.
+To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
-We have now found the unknown parameters \( z_{i0} \). These correspond to the projected coordinates and we can write
-$$
-J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2.
-$$
-
-
-We can show that the variance of the projected coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by
-$$
-\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2,
-$$
-
-since the expectation value of
-$$
-\mathbb{E}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \mathbb{E}[z_{i0}]= \boldsymbol{w}_0^T\mathbb{E}[\boldsymbol{x}_i]=0,
-$$
-
-where we have used the fact that our data are centered.
-
-
-Recalling our definition of the covariance as
-$$
-\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T],
-$$
-
-we have thus that
-$$
-\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0.
-$$
+
We are almost there, we have obtained a relation between minimizing
@@ -3610,7 +3559,7 @@ of the projected data.
We could trivially maximize the variance of the projection (and
@@ -3664,7 +3613,7 @@ For more details, see for example Geometric Interpretation and link with Singular Value Decomposition
+
For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.
@@ -3672,7 +3621,7 @@ For a detailed demonstration of the geometric interpretation, see Principal Component Analysis
+
Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
@@ -3727,7 +3676,7 @@ X2D = X_centered
-
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -3759,7 +3708,7 @@ variance that lies along the axis of each principal component.
@@ -3800,7 +3749,7 @@ We see that our training data after the PCA decomposition has a performance simi
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -3830,7 +3779,7 @@ X_reduced = pca
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -3842,7 +3791,7 @@ instances arrive).
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -3853,7 +3802,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
@@ -3882,7 +3831,7 @@ X_reduced = rbf_pcaLLE
+
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -3894,7 +3843,7 @@ these local relationships are best preserved (more details shortly).
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
diff --git a/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz b/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz
index 7d7ff64ea..ddd7e561c 100644
Binary files a/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz and b/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz differ
diff --git a/doc/pub/week43/ipynb/week43.ipynb b/doc/pub/week43/ipynb/week43.ipynb
index ec3fcba20..04536e6ab 100644
--- a/doc/pub/week43/ipynb/week43.ipynb
+++ b/doc/pub/week43/ipynb/week43.ipynb
@@ -10,12 +10,13 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Oct 23, 2020**\n",
+ "Date: **Oct 26, 2020**\n",
"\n",
"Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
+ "## Plans for week 43\n",
"\n",
"* Thursday: Wrapping up Recurrent Neural Networks and solving differential equations. [Video of Lecture October 22](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober22.mp4?vrtx=view-as-webpage)\n",
"\n",
@@ -672,7 +673,9 @@
{
"cell_type": "code",
"execution_count": 1,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -837,7 +840,9 @@
{
"cell_type": "code",
"execution_count": 2,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1091,7 +1096,9 @@
{
"cell_type": "code",
"execution_count": 3,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1351,7 +1358,9 @@
{
"cell_type": "code",
"execution_count": 4,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"# Assume that all function definitions from the example program using Autograd\n",
@@ -1552,7 +1561,9 @@
{
"cell_type": "code",
"execution_count": 5,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1860,7 +1871,9 @@
{
"cell_type": "code",
"execution_count": 6,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2283,7 +2296,9 @@
{
"cell_type": "code",
"execution_count": 7,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"def sigmoid(z):\n",
@@ -2382,7 +2397,9 @@
{
"cell_type": "code",
"execution_count": 8,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"# Set up the trial function:\n",
@@ -2449,7 +2466,9 @@
{
"cell_type": "code",
"execution_count": 9,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2807,7 +2826,9 @@
{
"cell_type": "code",
"execution_count": 10,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3335,7 +3356,9 @@
{
"cell_type": "code",
"execution_count": 11,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"# Importing various packages\n",
@@ -3366,7 +3389,9 @@
{
"cell_type": "code",
"execution_count": 12,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -3410,7 +3435,9 @@
{
"cell_type": "code",
"execution_count": 13,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -3439,39 +3466,11 @@
},
{
"cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- " 0 1 2 3 4 5 6 7 \\\n",
- "0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 \n",
- "1 0.0 0.070063 0.069533 0.067085 0.068151 0.069193 0.057333 0.058540 \n",
- "2 0.0 0.069533 0.069368 0.066521 0.067744 0.068951 0.056953 0.058268 \n",
- "3 0.0 0.067085 0.066521 0.068521 0.069618 0.070705 0.061297 0.062609 \n",
- "4 0.0 0.068151 0.067744 0.069618 0.070847 0.072071 0.062332 0.063755 \n",
- "5 0.0 0.069193 0.068951 0.070705 0.072071 0.073435 0.063371 0.064907 \n",
- "6 0.0 0.057333 0.056953 0.061297 0.062332 0.063371 0.056744 0.057998 \n",
- "7 0.0 0.058540 0.058268 0.062609 0.063755 0.064907 0.057998 0.059350 \n",
- "8 0.0 0.059802 0.059641 0.063981 0.065242 0.066511 0.059310 0.060763 \n",
- "9 0.0 0.061119 0.061072 0.065414 0.066793 0.068183 0.060681 0.062239 \n",
- "\n",
- " 8 9 \n",
- "0 0.000000 0.000000 \n",
- "1 0.059802 0.061119 \n",
- "2 0.059641 0.061072 \n",
- "3 0.063981 0.065414 \n",
- "4 0.065242 0.066793 \n",
- "5 0.066511 0.068183 \n",
- "6 0.059310 0.060681 \n",
- "7 0.060763 0.062239 \n",
- "8 0.062282 0.063866 \n",
- "9 0.063866 0.065561 \n"
- ]
- }
- ],
+ "execution_count": 14,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"# Common imports\n",
"import numpy as np\n",
@@ -3504,7 +3503,7 @@
"\n",
"\n",
"# Making meshgrid of datapoints and compute Franke's function\n",
- "n = 3\n",
+ "n = 4\n",
"N = 100\n",
"x = np.sort(np.random.uniform(0, 1, N))\n",
"y = np.sort(np.random.uniform(0, 1, N))\n",
@@ -3726,7 +3725,7 @@
"source": [
"* Center the data by subtracting the mean value for each column. This leads to a new matrix $\\boldsymbol{X}\\rightarrow \\overline{\\boldsymbol{X}}$.\n",
"\n",
- "* Compute then the covariance/correlation matrix $\\mathbb{E}[\\overline{\\boldsymbol{X}}\\overline{\\boldsymbol{X}}^T]$.\n",
+ "* Compute then the covariance/correlation matrix $\\mathbb{E}[\\overline{\\boldsymbol{X}}^T\\overline{\\boldsymbol{X}}]$.\n",
"\n",
"* Find the eigenpairs of $\\boldsymbol{C}$ with eigenvalues $[\\lambda_0,\\lambda_1,\\dots,\\lambda_{p-1}]$ and eigenvectors $[\\boldsymbol{s}_0,\\boldsymbol{s}_1,\\dots,\\boldsymbol{s}_{p-1}]$.\n",
"\n",
@@ -3737,7 +3736,7 @@
"## Writing our own PCA code\n",
"\n",
"We will use a simple example first with two-dimensional data\n",
- "drawn from a multivariate normal distribution with the following mean and covariance matrix:"
+ "drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below):"
]
},
{
@@ -3756,8 +3755,8 @@
"metadata": {},
"source": [
"Note that the mean refers to each column of data. \n",
- "We will generate $n = 1000$ points $X = \\{ x_1, \\ldots, x_N \\}$ from\n",
- "this distribution, and store them in the $1000 \\times 2$ matrix $\\boldsymbol{X}$.\n",
+ "We will generate $n = 10000$ points $X = \\{ x_1, \\ldots, x_N \\}$ from\n",
+ "this distribution, and store them in the $1000 \\times 2$ matrix $\\boldsymbol{X}$. This is our design matrix where we have forced the covariance and mean values to take specific values.\n",
"\n",
"The following Python code aids in setting up the data and writing out the design matrix.\n",
"Note that the function **multivariate** returns also the covariance discussed above and that it is defined by dividing by $n-1$ instead of $n$."
@@ -3765,8 +3764,10 @@
},
{
"cell_type": "code",
- "execution_count": 21,
- "metadata": {},
+ "execution_count": 15,
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -3775,7 +3776,7 @@
"from IPython.display import display\n",
"n = 10000\n",
"mean = (-1, 2)\n",
- "cov = [[3.0, 2.0], [2.0, 2.5]]\n",
+ "cov = [[4, 2], [2, 2]]\n",
"X = np.random.multivariate_normal(mean, cov, n)"
]
},
@@ -3821,14 +3822,16 @@
"source": [
"When you are done with these steps, print out $\\mu_n$ to verify it is\n",
"close to $\\mu$ and plot your mean centered data to verify it is\n",
- "centered at the origin! Compare your code with the functionality from **Scikit-Learn** discussed above.\n",
+ "centered at the origin! \n",
"The following code elements perform these operations using **pandas** or using our own functionality for doing so. The latter, using **numpy** is rather simple through the **mean()** function."
]
},
{
"cell_type": "code",
- "execution_count": 22,
- "metadata": {},
+ "execution_count": 16,
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"df = pd.DataFrame(X)\n",
@@ -3876,21 +3879,11 @@
},
{
"cell_type": "code",
- "execution_count": 23,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- " 0 1\n",
- "0 2.983691 1.981141\n",
- "1 1.981141 2.466534\n",
- "[[2.98369133 1.98114104]\n",
- " [1.98114104 2.46653391]]\n"
- ]
- }
- ],
+ "execution_count": 17,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"print(df.cov())\n",
"print(np.cov(X_centered.T))"
@@ -3906,31 +3899,11 @@
},
{
"cell_type": "code",
- "execution_count": 24,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Centered covariance using own code\n",
- "[[2.98369133 1.98114104]\n",
- " [1.98114104 2.46653391]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOy9fXRTZ3q3ez17S7axsRUja2xkTBxLGBN7TBy+EgcGBuYlpCRtSDtnpiSckg9CWM3hfYe+M22Sc9Y6a52VpNNp6TqsmUMIw0AXSdp520I6gYbQgUAgJHw4Dg4kxtiOibGMxxaObPwpae/zx9belmzZGGy+7OdaKyuWtLW1JaF73/v3/O77FrquI5FIJJI7F+VWH4BEIpFIRoYM5BKJRHKHIwO5RCKR3OHIQC6RSCR3ODKQSyQSyR2O7Va8aEZGhp6bm3srXloikUjuWMrKylp0XXf1v/+WBPLc3FxOnTp1K15aIpFI7liEEBfi3S+lFYlEIrnDkYFcIpFI7nBGJZALIe4SQvybEKJSCPGVEOLB0divRCKRSK7OaGnk/y+wT9f1PxNCJADJo7RfiUQikVyFEQdyIUQa8D1gNYCu671A70j3K5FIJJLhMRrSSh7QDGwXQpQLIX4thEgZhf1KJJLbjDcO13CspiXmvmM1LbxxuOYWHZEERieQ24D7gc26rpcAHcDf9N9ICPG8EOKUEOJUc3PzKLysRCK52RRPcfDiO+VWMD9W08KL75RTPMVxi49sfCNG2sZWCJEFfKrrem7k9gLgb3RdXz7Yc2bPnq1LH7lEcmdiBu+n5k3lrePf8MuVJZR6Mm71YY0LhBBluq7P7n//iDNyXdcvAfVCiOmRu5YAX450vxKJ5Pak1JPBU/OmsulgNU/NmyqD+G3AaPnI/w/gbSFEBXAf8Noo7VcikdxmHKtp4a3j37B+sZe3jn8zQDOX3HxGxX6o6/rnwIB0XyKRjC1MWcWUUx7wOGNuS24NsrJTIpEMm4qLgZigXerJ4JcrS6i4GLjFRza+GfFi5/UgFzslEonk2rlhi50SieTGIr3bkqshA7lEcpsjvduSq3FL+pFLJJLhY+rQ0rstGQyZkUskdwA3y7stZZw7ExnIJZI7gOv1bl9rYJYyzp2JDOQSyW1OtHd7w9LplswynGA+nMAcHexNGWftzjJWbv2UF98p5+HCzLjHJLP02wcZyCWS25yReLej9fWN+8/FLd7pH+wBgmGNYzV+npo3lcdmumWWfpsjfeQSyThg4/5zbDpYzfrFXjYsnT7g8ehGWNuP1QHwdGmutbAKWI9vPfI1G5ZOY80CT8zzKy4GeGGhZ8C+r4c3DtdQPMURc8IZ7de4E5E+colknDIcfT16MTUY1tiyalaMjANYjy8rymLzodprztCvRa+XWv21Ie2HEskYZri9Ucxg/5DHSUVDn2RjSjPvnfbxwdkm62SwblHesOyQ0Zm1GZzXLcojrPUFazPjj0ZaLq8NmZFLJGOI/llvxcUA6xblWXp6PH09Oti/veYBtqyaNUAz/+BsU8xi6+ZDtSzMzxjSDvnG4RpUBWtfpZ4MHinK5NW9lZy71HbVZlujabkc67ZKGcglkjFEf0mieIqDzYdqYySJUk9GjM58tcXUeI+vW5THvjNNQ8o15mub2ftPflvOO8frme91srvcd9XgfKymha1HvmZFiTvmNa4nAI91qUYudkokdwjDXQB8aVcFeyoaByxWRm83ksXE/nJN/9vxtvW6UjhR18p8bwZfNrZdVS4xn7duUZ51Moj+//XILGNhspFc7JRI7nCGm1U+NtNNMKxZkgQwYLuRZKjXYocs9WSwMN/FibpWpmel8nF1C+sW5V3VD2++xpoFnigpx8XG/eevOwCP5clGcrFTIrlDuJYFQLtq5GhvHqll+7E6tqyaFbPdSBYT42XspZ6MuM/deqSGd8sbWFGSzb4zl1g5L4fNh2opdBtXAw8XZvLead+AKwNzn+b/zQC8frH3ugNwf/fOAx7nmAnmMiOXSO4grpZVmpn1llWzeH5BHt1BjWBYu+q+FuZnDAimpg7df6Fw9fYTvLK7gjcO11iPbT1Sw+rtJ2Kee6ymhY37z/Py8gL+8Uf3sW31bHZ95uORokwre39spps9FY28tKsi5vijrwyuZp8czkLmSKpj7wRkIJdI7iCuFtRMSQKwtrOrCu+d9g26rxUlbt4t97H1iBGYtx6piQmmqgLP7jhlvVb2XUm8fbye+ssdXPB3sPo3J3htbyUPeZ0cq2lh7c4yLvg7qLgYYNvq2VbhUKkngw1Lp7HrM98ACWdPRWPcytPhBODhyERjfbKRXOyUSG4R17rgGG+R8dkdpwZUWW49UsPG/efZtnr2oIuR/e/beqQmEowz+Li6hZeXF7BmgWfAoqMpwzxSlMk7x+uZnjmRyqYrJKiCFxZ6rKrQ/lJOvPdh7uvhwky6g2F2l/usylPzcwCG9RmNhYXM4TDYYqfUyCWSW0R0QUz/gBuPeFnlhqXT2Lj/vKU5m3LGhqXT4maf5n3997VmgYcvfe3sLm9gRYmbzYdqae8KxQTF9q5QTJn/+aYrnKhrpSBzInWXO9l0sJoku8JvVs8ZMoj217xTJ9h4bW+lZTNMnWAb0pkST48fLR39TkUGconkOhiNXiDXuuAYb79rFngodDti9mFm4v1f62qe7d9/1USpx8nhqhYW5rtigmJ/Scff0cPJulbm5qZzsq510H3G+zyi92Vm8C8vL7CcKa/treTl5QUDjneoz7x4imPMLmQOB6mRSyTXwWgVmIyGJe569hF9/KauDfDiYi/rFuWxu7yB+V4nbx3/xtLM1y3KIznRxiNFmbx9vJ6V83L4H/8tH7sq0AGbAooQrN1ZNkBnN+mveT9aPBmAQreDp+ZNZXd5A4+XZBNvfXawz9ysHh2rC5nDQWrkEsl1Mhq6rLmPgqxUyi608j8fzrf07mM1Lbx32sfdzpQhs/xrOY7orNbU2DPTEvF9282OZ+YAsHZnGV5XCpWXrvCgZxKHq1pYlJ9Beb0hx/x8XyUd3SE0HXIzkjlZ18ofz5zMGV8bVZfa6Q3rgM7fPFJg9VQxX+/Nj2p5yOskrPVdYZjv84OzTcMuFIrezszIx0OnRFkQJJGMMiPNpl/aVcHanWVGBrnYiyLg1b2VPLPjBMdqWlj9m5P8x+e+AVa84djqXtpVYexj+wm2Hqmxtr3g7+DZHSf5k18dpdSTway776LO34n7riQA3jvtoyeo8UfFk9mwdBpHq/1ous6BymbuyzGOo7a5g4Zvu8jNSOaTmsusX+Ll1RXF/PWyAnQgrOkUZKWx+VCtlS2bGfpDXicb959H7Rd59lQ08nBh5lUz6nif+QsLPXGlpLEWxIdCBnKJ5Dq53vFr8TAWLvMBOFzVwjM7TmJTBaoiYl7PlCtM73T0omX0bTACaPZdSby2t5JXdlfw7I5T/KGtm+6gRtWldn7y23I+rvaT60ymzt/J6u0n+feyBn44O5tNB6rZdKCa5d/NwrxoP3SumdW/OUEorGFTFZ5bkGcttm7cf461O8uwqwrrF3tpDHRbThdT916Yn8HmQ7VsWDotpg2uaY18bKbb+iwGswaO5mc+lpDSikRyDZjSBBDjMDGlgWuVV/pLBQvzXewubwBg/WKv1XZ2RlYqFQ0BtqyaRcXFAKoCmw5U82jxZF5/oniA5RD6eq7MnOLgaLUfxwQbga4QJTkOcjNS2F3uQ1Xgf5udY9n/VEVgVwWKEOi6Tk9IQ+sXImyK4Hv5xmt8UnOZZUWZ7C73xThWzPe1MD+D3eU+5uamc6KuNcZeeK2y1LX0eBmr3HBpRQihCiHKhRB7RmufEsnthrng9t5pX0zm+9hM93UVmFRcDFgOkYX5Gfz+qyYSbAo2RViOjqfmTeXjGr9VoVk8xcGmA9WEwhp7Khr5yW/LeW1vZYzlEPp6rhyt9uN2JBHoClmv+R+fG0Fc00AR8Puv/kB6sp2wptMd1OgNhekOGkFc9DvmkKZz6FwzH1Y288T9bg5XtfCQx2m1BYDYDokrSrI5Wdca08XwemSpsV7UMxJG037434GvgLRR3KdEclsRbRl0TUwckE1ebbiCiSmDqAq8W97AfK+RudoiGfFPH5nOpgPVPPdPp9B13Rr4sHZnGU+X5gJgUxVmZKWyu9xHoTuNQnesQ+SsL4CmGRm0L9CNIkDTIRzJsFUBK+fl8PbxepITVObmpnOgshmAUJRrJN41u6ZDkTuN9880xc2QgRgZxbQXmi1t78txcDKSoZt2QWDIBcpr6fEy3hiVjFwIMQVYDvx6NPYnkdzOXGs2OZRtzgxyZRdayXUmE4roGIVuB+uXeOkNafSGdRYVuNiyapbV1fAHMzKZOimZExEv9/k/XOEvtvUtbG49UsOreyvJmJhg7TPBpuB1pVjHFdaMTNyuCtyOJA5WNrOkwBWTgffPxqPv/7KxjftyHHEzZDN7DmvEdDEMa7BuUR4fV/sBeMDj5JcrS1i7s4y1O8vGTH/wm82oaORCiH8DXgdSgf+p6/qjcbZ5HngeYOrUqbMuXLgw4teVSG4FI9F3B7PNmcORV5S4SbKrfHC2CU3Tae8O8uO5Obx/pon7chwcqWrBpgq6gkbKPN/rpOzCtzxxv5u3j9cDsKIkm70VjQgBvSENRRH85SIPvz76Nb0hDVURTEhQ+bYzCIArNYHm9l5Kchx8Xh+wMnAFiNduy6aAqiiENR27Kni8JJvXnygesF30lUj02sKbH9Xy/PfyOOsLsHH/edYsuIftx+osvV8yODdMIxdCPAr8Qdf1sqG203X9TV3XZ+u6Ptvlco30ZSWSm050QynT7rduUV5MQ6nBOvFVXAwMapuLnoRzuKqFx2a6eWreVL7tChLW4V9PXeSRokwOVjYT1HS6ghqKgESbwumLATYsncb7Z5pYUmD8rnaXNxDWNHRdRwd+PGcKD3icTHYkEdZ0/mxWNs6UBOv4mtt7cU1MoDwqiGelJcYN4gA6gp6QhuiXrve3RkZfiRRPcVhZ9/PfywOMq5FlRVlsOljNd7MdlmtlsP1JBmc0pJWHgD8WQtQB/wIsFkK8NQr7lUhuK4qnONi4/zzrFuVZAdjUgaMbPA0mo8SzzZmPb1g6jcNVLdyX4+DZHSfZeqSWBJvx8+wN61a2bTJ1UjIJNoVHiydbcsV30pJIUI3oGtYhGNZ5cl4OvzvdyNqdZcy9ZxIr5+XwLycuUn+5k0Rb38+/+Uqv9beqCC619RDlfERgnDgUYfjEk2wKwbDOgx4nrz9RHLeyNXo94dMav3X/pzV+q1L0cFUz6xd7+SKi/4/VUWw3mlG1HwohFjGItBKNtB9K7lTMashlRVkcrmoe4OF+YaFngIzSfzxZdNfC6MrHrUdq+Lt95wiGDcniz2ZNobKxjfL6+K6MaCvf2p1lhMIaQgi6g2HLMjjfm8Hpi98CWKPf7stx8Gnt5bj2wsy0RJrbewCs+9OT7bR2BslKS6SprYdEm0J3SKMoOw3ft908NW9qXGnElFM+rfFbfVvqWzvZXe6zrj6iPxOzTUD0iDq5kBmLrOyUSEaBUk8Gy4qy2F3ewMJ8V4xbw8we+y+Gmgt+8boWRpevbz5US35mKgCaplN/uXPQIK5G7Immlc/rSqErqNETCqPrRgAHOFrdwg9mZFKc7bCO5zer5+JIsg8I4grQ1NaDpvcFcQG0dgZxTUzgUlsPaRNsdIc0bIrgTEMbbkcSmw5WEwxrPDbTHSMtmXLKr49+TanHya+Pfm0F8X1nmqwrG/Mz2bJqVsxxyiA+fEa1+6Gu64eAQ6O5T4nkduJYTQuHq5pZUZLNu+UNgG5llm9+VMtZX4BCtyOieWez+XANUyclW5m6ucAZ1mDb6tlW5r71yNekJ9v5uqWDFSXZ7C5v4Gi1f9DjUAC7ajSompObztnGdjIjGXOhO42F0zP4tNaPDvzH5w3YFEGSXWH7sToe8DjJciTS2NYNYBUKmZq4wLAcJkUyb4EhvaQkqAS6QqQn2/nVk/fzF785wRlfW8QyaeSE0a15wZBhOnvD+L7torM3THKCyg9n5/DD2Tms3VlGbXNHTBb/1aX2cdvBcCTINrYSyTDpX0lY1dRmZZilngzO+gK8urcSuyr42bLp/Pz9c4Q0nZrmDl7ZXcH7Z5p4pCiTv//gHN/Lz6B4isPK3IvcaZzxtQGQnKCQaFPoCcUuN6qizwMO0B3UCIU1jlS1YLcp+K/0ogg4d6mN1/a28f0CF60dvZyuD9Ab1lkyLYNPai+zcuvxmP129IStvxNUQViHGVmpnPG1WUF9YqLKlZ6wlaG/vOsLgpGDmXvPJL5f4OLZHafYtnq2pYvPyEpFVYR1YkpQBQ/kTeK9076Yhc3oplnmZ2tWtEp5ZXjIEn2J5CqYWm+0XXDrkRp+sa/KSl/NzoF/8ZsTBMO6VZIOWEF6fmT6zsrIAqTZs+QHM77D7nIfdkWgoxPS+rJiE7OYx5FkI9BtlNkvmGZUhBrBVyesGZLK0eoWFAF/PjeHfz5Rj6aD25GEL9A94L2l2BU6IlZGc3EzLyOF6uYOcp3JNHzbZQVsRcA9GSlcvtJLa1cQNcrWGArr/HRZvtXVMNpOebiqxdLRQ2ENHWM49JZVswCjMvbhwkwem+keFx0MR4LUyCWSKIYzsNfElAuig/hreyv56bJ8djw9ByFg9W9OsHZnGf/0zFympE/gRF0rOekTWFHi5oyvDWeKnaPVLdztTObVFcWsX+KlO6jR0R3i3XIfE+wKSQkq07MMjVwnthhH041AGugOketMpqKhja1HallR4iYYNoJ4TvoEjla3WF70t4/Xo0Q8gqaM0h8ziIMRwFUhqG7u4Ml5Ofg7eglHieiKgIutXbR2Gf7zsKbzu9M+OnvD2NTY5l7Rs0DXLcpjw9LpbFk1Cx2sgdCme+WXK0t4/Ynicd/BcCTIQC4Zl1zLYIiKiwGrtHzj/nNs3H+elfNyCGtGsPmj706mN6wzdVIyZ30BLrZ2AVDf2sW+M5dwTLDh7wiSYleo83fy7I4T1DZ3cFeyHQ0jaOe5JvLHMydz7tIV63V1IMne9xPVdBAC6vydCAw/+buf+6ygX9/axcRElT0VjeRFKjhDmo4QcLUL75z0CVQ3dzDVmcwEu4Kmww9mZFqLnikJCiENekJajC2xzt/JfG8GK0rc/O375zjxdV9wnp6VZpXmv7SrgrO+AHZV4aFIOf6mg9XMyEqNCeDXcoKV9CGlFcm4ZbgVmn2d/FyRmZbZlvUQjAnzs+5O52i1EYCSE1QezJtk9S0Bw2US1vQBEocCiMhjJh5XCjXNHTHHEK2Pm3FU7/dY/32btsGhUIQx1Sek6RRlp6Eqgnsnp7GnopGOnpB1JaDpfYui5qJq/2OxKfDD2TnkuVIIa6Aq8HG1n+e/l8evj9TyYWUzLy8voNBteOW7ghoT7ArbomZ8yg6HQyOlFYmkH8PtmVLqyeC+HAe7yxtwJNnYXd7AukVGifmzO06RfVcSn9T6SU+2A3D/1Ls4Wu0nMy0RMILdwvwM7KoYoFPbVIUfz5liZbmKgJrmDlQRsxmZaUnW3zqx+rkZ4P0dvRS5+3rWXS2IA1aTrgSbwpkGY8JPniuFYNiwJuY6k0myq6gKBLpC5KRPsIK4TRExxxLS4A9t3Ww+VEv95Q5e21tJ9l1JVFwMMPceJy8vL2DTgepIr3WFV5YX8HhJdsyVUXQR0cb953h2x6kYmyLIDD0eMpBLxi3DHVJwrKaFk3WtJNgUAt1GK9i//6CKX3xQRalnEjXNHQh0vu0MMj1rIker/ei6TmtHkBUlbiYm2fi09jKhKMuJGaczJiaw6zMffz43h8zUREvKCOvESBjxFir7EwxrlvNlOAgM6SUU1pmbOwmbItB0+M+KRkJhnczUROr8nXQHwyTaVHKdydRHZCOAgoiebzIxUeVAZTMpCSrvRGZ6vn+mieIpDl5Y6GHNAg/fzXbQHdR4ujSXNQs8vP5E8YBWtNEn2GVFmZY0Y84XNSUwGdD7kNKKZFwS7xLerLY0Z2aa25lzJn+xr4qwrqPrOpoOrokJtFzpJcmu0BvWUYVRTm9TjOzU40rhwF8tioxtOxGZZdlHkiroDut4XSk0tfdYjo6eoGZluWlJNtoiJw9T4uiPKgBB3IHF8TAdMQqQ5Uii5UoPvWGd9GQ7zpQEqps7mGBXeNDj5NC5ZqsnuaCviVZ/GcfcZ7TM0x3SeLgwE0VAw7fdPP+9PEui2nfmUswQjHjfTXRl7KYDRtFRf7fLeJNcpLQikUQRb0jBhqXT+Lt952JmXJrj0jbuP89Pl+WjRgXT5iu9RofBsE5Y0wlGHjAfT02y8cbhGv6vd89YAT6a7rCOgiGldPeGsakKf3p/NnZbXypuBvHo/fYnrA8/iEOfFKIBPaGwdaXQ2hmkurkDmwKPl2Tz3II8y/Wi0xfES3IcMS6YIneatU/zXOULdON2JFHZ2Mbbx+tRBTH9VTYsnRZ3Lme8GaSbD9XygxmZcd0u4ymID4UsCJKMS+LZ2sxM/LW9lXzpa+dwVTPrFuWxcf95NiydRqHbQaJdJRTuq4LUdHCnJeILdKPrkGxX6AwaJexf+tpZXjyZby53AiAGuMP7gqNQwOtK4e3j9YP2AL8R+DsG6ughDeovd7Jq2wkSVMGU9AnU+Tutx6Nb3Ra50zjra8PrMrzn0Zm6KfM8OS+HXZ/5LJnEDMCFbsObHx2MKy4GeLgw07ptThr6+w+qKPU4+eybVqtviwzifciMXCKJYs0CD49HKhG9rhQ2H6pl2+rZFLoNu+Ifz5wc097VkWSzApeASBA3tOepkybwcbWfny2bjteVYmXs8egN6ZTXB0i0CSal2G/smxwGR6v9hCMtc6ODOPSdinKdyTw6083Lywv4uqUTtyOJxkA3873OmJPR8mI3axbcw+5yX8yicjyf+AsLPTw2021l68dqWth0oJoEm8L3C1zYVcVqNSAHL/chA7lEEoXZS8WszFyYb4wSM6WYM742ElSjb4nAKNAxrd46Rom7WVn/tb+Th7xONh2o5mt/J44JV78A7gnpcbPkW41gYLC42NrV57sXMCFB5fESt7HYC0zPmgjAL/ZVDrmo3N87bmbhz+44xa8OVgOwfomXzYdq2bJqFr9ZPYdHiyfHlWbGK3KxUzKueeNwDRf8HVbvjxffKeeRokz+rawB73cmctbXxpPzcnh1RTEv7arg3fIGhDB6hpysa+VKd8jKUJPsCt1BDbsCZsGkwPCQ5zqTqe7nDR8LmAuwT87L4V/LGghFbIsJqiDRruJ1pVBeH7A+w3i+8MG84wvzjTmm6xd7SU60DTr3dDxVf8rFTsm4YrgVgsVTHOypMAYvvHfaxyNFmbxzvB5FwCvLZ/BkZDixuQBq2PU0Dle1WPMvzeHx3ZHoHVX1jo5Ryl7d3DEq2ndif4P5LUbTjR7my4vdEHHzpCfbI2sJGhUNbZTkOGj41pCfzGz7zY9qrX309473LYq2WFl8/yBuPm88BfGhkIFcMiYZTgn+6u0nOOsLWHa23eUNvHOiHiGwqg1fXVHMK8sL2Lj/PK6JiQiMKkhzNNriAhcL84ceXWhWPY7GtW9P+OZfQcfDkaRafze19fCT336OqghsiuF+mZGVik1VWJifwYXLXdZ4N7PvunnbJNo7vjDfZS2Kms4VKaMMjQzkkjueeNk3wMOFmTFZnll4Ym77kNfJa3sr2VvhozhSqKLrUBrVB/uNwzUUuh2sWXAPmw5Ws7x4MoqAS2095DqTOVjZzMGoUvzBCF2DPfBOINAdJiWqD0xTWw89IY0Em2o1DZs5xcHce5wDsu14tsHo4qx9Zy4NGDrRv2hIEou0H0rueKKHGfTvz+GamDjArmY+tmaBh7qWjhjLn6rA6UiwL/VkWFNueoIaK0qy2Xem0QrK/d0cdxIDjZDXTnTnRDBklt5QmEBXkKLsNI5W+7nbaQzVMLPteLbB/hq52Yu80O2ICebSbjg4MiOX3BEMpXn3705oaqzvnfYNcEv012N/d7rRCmrTs1LZ+ew8gJhBwAA6Ou+dbqArqGFXYnXq20u1Hh7XE8SHep/mYyENnCkJnGloY0mBi12f+dh6pGZI10q84iyZgV8bMpBL7giG0ryLpzjYfKiWhfkZEY01g00HqtlT0RhXZ43WY83GT9OzUqm61M5ZX4D1S7zkuVKouBig4qKhoednphLSDK17Xt6kmGO7PVTrG89g79Mqz4+c4IzWtk62rZ5rzSYdSu9+YaFHLmSOEGk/lNwxDNV21hz2MCc3nZN1rXy/wMVzCwZ2zau4GOCCv4M9FY04UxKo83eSoArmT8ugtaOX8voAioCX/qjAqjy84O9g12eGte42WWu85dgUwXez06zh0KoirB40jgk2VEWxsmpVwZocBOPTNjhaSPuh5I5nsLazphPi8ZJsTtS18nhJNp/HmT5vBpU9FY109oZp7w5aHQAPVjZbQSnJpvDz98+xdmcZWz+q4Z9P1NMT0mJayY5HoqUVz3cm8mVjOzYF7IogY2KC1Qsm0BXikSJjoVlVYPOh2hi3kJltyyESo4cM5JI7hsHazlZcDHBfjoPff9XE+sVeq0fKe6d9MUGheIqDjfvPs36Jlx/PmYK/I4iIFLQkRTWq6gxqhDSd9u5QTJXlcFrJjmV0jAIgAZy7ZATxsAbfy8+gqa2H3IxkwOgZ887xeu6dnGbJKvEWKq9lSpNkaGQgl9wRxOuK9+yOU2w9YgxGPhkZdJw6wcbDhZmWRt4/E3y8xM2mA9U4UxKtqkRFQHdIJy2pz8R1Jy5g3mgExudlVwUpCSodvRrfSUvkQGUzRe40LrR08uS8HC53BnkoMgR6zYJ7hhzYMRxrouTqyEAuuS242mX2YG1nN+4/z8/3VbJ+iZf1S7y8treS7qBGKKyR50rhzY9qrapMgMdmuunqDbPpYLUVrM1gHt0yVkrhBtGFpFZGLgQdvWFynck0tfXgmmjnjK+Nx0vcvLqimHWL8ii70HrVgR0w/ClNkqGRgVxyW3C1y+x4zoY1CzxsWz2b2uYO/mF/FZsOVFudC3Xgr5cV8JDXyat7K1nxq6Mcq2llTqkAACAASURBVGlhb4WPUETMjV64HKIx4bgmrBsDMEyS7So9IY37chwc+un3KXKn0XwlSE76BPadaWLrkRqrY+RwqjKHO6VJMjQykEtuC673MrvUk8HTpbl0BzW6g2HeP9NIkl3BHmmAsmaBhyfn5RiNm7Ye5+3j9XhcKfIf/jXQHXXGu9IbJjMtkXOXrvDK7grO+tooyk6jvrWLgqyJ/OKDqgEzNh8uzIzrCY8nl8lS/OtjxP+ehRA5QogPhRBfCSHOCiH++2gcmGT8cT2X2WZGt6LETTCs0x3UeH5BHltWzbI09FdXFFOQOdGSS75u6UADstISrYZX8Zhgl+E+GrMOqqmth4Ksibx9vJ6HvBn4vu3mleUFVEeGRtdGujyagfqxme64VkNZCDR6jEaJfgj4K13XPxNCpAJlQoj/0nX9y1HYt2Qc0f8y+4GonieDbW9mdO+d9jHBrqAD24/V8YDHaWnodS0dnGu6gjPFjr8jiKbDfK+TBJti9UmJbj1r0tX/jnGOrkNygsr0zImU1wcocqdxtNroULhmgYdCt4P3Tvv44GwTronnBnj9+xMvuMtS/OtjxIFc1/VGoDHyd7sQ4isgG5CBXDJsBuu3MVQgMDM6gA/ONrFt9RwA3jvt48V3ynm4MJNSzyTePl5vjSQzOVrtx+3o84XLmH11dIzy+8pLV6zxbf1PuqWeDFwTz8lxbDeZUa3sFELkAh8BRbqut/V77HngeYCpU6fOunDhwqi9ruTO543DNdc0OCB6e/NvwNr+WE0LP99XyRcXA0ydlEydv9PyPacmqbR1hwGjl3ZTW8/NeZNjhCJ3Gr5AN+sW5RHWYpuWvXfax7+XNbC8OIvDVS3Wifbn+yq5d3Iarz9RbO1HVnheOze8slMIMRH4d+B/9A/iALquv6nr+mxd12e7XEP3b5ZIhuKNwzWc+NpvNbZ6YaGHs74AT28/yQW/oc9WXAzwaPFkVEVQ5+8kY2ICIc3IKtu6w7gdSSiCcR3Ek69zDeCMr41JyXarYvPNj2p5pCiTn++r5N3yBmyqYG9FI+nJdp7dcZK/+M0Jqi61s6eiURb/3CBGJZALIewYQfxtXdd3jcY+JeOLa6nyMwuAQmGNtTvL+MlvP+e1vZUoAmtkW/EUB5sO9HnFW670xuzDF+ge15ZDAXRfY5P09GS79XlWN3dY7pTsu5J4+3g9CmBTFR7Mm0RvWOfrlg5jnUE3BnVsWTVLFv/cIEbDtSKAbcBXuq5vHPkhScYj12I/LPVksGXVLGyqQmdviN3lDdhVYU31MbeZmGhD1/vcFmA0e5IYVybXqqq2dgYt58/EBJWwZjQr2/WZjyfn5fB5fQD3XUkcqGwmJ32CdaJct8hj6eey+OfGMBqulYeAVcAXQojPI/e9rOv6f47CviXjiOgferyFsmhdvNSTwdRJEzjraweMgp6zvgDvnfbxZWMb905OIystkcZIfxRVGMUtIU23SvPHM1cbLGFXBcEhWj1e6Q1z9HwzH1f7eXl5AWsWeDjfdIUTda04U+zUt3ahKmBXFctFBFyTK0kyfGQbW8moc60Ll9HbxGtTG72YaWbqeyt8vH28HjCm+qhC0BvWSbQpKAKCYd2q4JSMPjnpE6hv7WK+18lbzz3AMztO8GFlM1Mi9ysCEm0KD3qclgxmUxW2rJo1YIqTDObDR7axldw0rkXvNnusvLSrgrU7y/jlyhIe8Dh5uDCTp7ef5JkdJ6z9AdyX4+B/33bCCuIlOQ4SbaoVtHtCGklRtyWjjyKgvrWL6VmplF34lld2V/BhZTN2VdDU3oPXlYKmQ1jTOVnXyvolXvKzUnm0eLIs/rlByIxcckMYaghEvO3uyzEWMM3BEHNy0/m42o+OUbwD8GntZbp6w5YkkJmWyB/aeqxWtFlpiVyKuFCSbMo1L+aNd4rcaZzxDTCcxSXXmUxbd4h1i/L4+fvn+NGcKWi60eu9ONvBybrLOCcm8sz8XMuiWHExYP1fWg6vD5mRS24qw13YMjOzz+sD/GBGJh9WNtMdDPNp7WUSbAo/nJXNwcpmPqpqoTMqiANc7ui1gniiTeFSWw8JqqDInSaD+DWiCPiysQ1VARFZD07qZ08scqdZf6cn23m4MJPNh2r560emkzMphdefKObp0lw+rvGjKIJn5uey+VAtqoI1ZGIklkM5iGJwZCCX3BCupaudGfR3lzcwJzedYFhH03V+MCOTd47XU5A50ZJKonujGNsZE2p6IoFbCMFXjcPLKiUGCvDnc3MicoixEFrkTqM7qJEa6dEuBHzV2M6T83JIUAUVDW08NtPNL1eWWGPcor9zu6qw6YAxP/W1vZUszHex+VDtiDRxOYhicKS0Ihl1+i9kXW1hy3x8Yb6Ld8sbuNedRlVTO8GwzvSsiZy7dAVFGHY5HSMbbO0MDnxhkI6UUcB0tHhcKdRf7kKIvj7k9+XcxRcNRrFV/yrN/t/5MztO0h3UmJubzok6oz/5hqXTR3Rsw5XsxipSWpHcNIbqahd9efzG4Rq2Hqnh6e0nuXvSBA5XNbNyXo4VxI2RYlfITDPmQSbYFBJUMWgQBxnER4oi+myJ9Zc7IXIC3bZ6Ds/Nv4djNX6eLs2NCeIw8DsHw3pY6E7jZF0rK0qyR6XfuPSix0cGcsmoE28IhDlw94K/g7U7y3hpVwWqAn/7/jmCYY2KhjZKchz8W1kDYU23ssKJiSpNbb3YVYGuQ6E7LZId3pK3Ni5IsivYFOgN6/SGNF5YmAcQVyozT8zR3/nWIzU8u+MU65d4aQx08/LyAmuO6kj7jctBFPGRgVxyUzFL6N8tb+Af9lchMHTunEkTOFDZzF3JdjQdbKpgvjeDKz1hozJTh58uy0cDkuzqNVclSoaHpoNjgh1zrTjBpvDGYeOqKd4AiHi69cb959mwdBphDX65soQ1CzyWlj4Sy6EcRDE4UiOXjIjrKf6J1k8BXBMTaL7SS1qkK2GRO43vpCXyYWUz2ekTuBgpMPmT+7LZd+YS2XclUR0ZXiC5MSgCXvqjAj6p8XOwshlFwFvPzbO+5+jv+Gbp1tdbaDaWkBq55IbQPyMzC3uinQRXs4g1X+nFphhdCVPsCmd8bRw614wOXGztIiEyM3J3eQOZaYlWEFelvHLD0HT40tfGybpWJtgVFk13xWTSplRm/m3q1gVZqQP2NVoWwaEku/GODOSSEdG/2dWeisaYx6MtYm8cruGZHSd4dsdJ7KpCUXafL9m8lO8IaghiFy1Nm6EqoM7faenjQ7QCkVwnKQkqqiJQFdhd7iMY1ti2eg6/WT3XCpj9/dzHalrYfqyOUo+TLxoCVnth8zFpEbzxjEbTLMk4p3+zK3O6T7xL7X/Yf45gWOfJ+7P511MNll3QrgiCkejdPz6bt83ArevxR7NJho/ZRAyMjpAhTbdsnUsKXByIjMCLR/QgCYC1O8sAeHGx17q9dmcZT5fmjkuL4K1AZuSSEWFaCKOdBGd9AWZkpQ6wiJV6MvizWVNITlD57cmLhHUdVQhynckENR1H0vDzChnER0ZY75OmQppOZloi30aC+MHKZhJUwUMeJ3ZVicmwIfYq7FcHqwGsZlhmi+HibIe0CN5EZCCXjAhVgdf2VrJuUR4blk5n3aI8XttbyWfftMa1qj02081z8+8hpOmENaOCs87fSa4zmUB36KqvJ5D/aEcLMyNPsis0tfVwrzuNlo5eVAUS7Sp/udjLllWzAGMOarTWbV6FfRzxlfcP1l9dapcWwZuI/E1Ihk28Xhe1zR18v8Aov964/xybDlSTZFd4vCQ7rlVt7c4yfn30a2sBMxyRVVquGM2uBEbDq8HQAZmMjx6OCTa6gxqqIvjmcicKRpn+H8+cbHWcNIN5tNY9mJ9bWgRvDTKQS4ZNPM/wu+U+HvQ4LY38u9kONizN525nCjCwXWlY0+nsDTP3nknWtJ6wpnOlxxiGrHPtI8gk10+gK4QijO/AfVcS5fUBVs7L4dUVxTxcmMnanWV8WuPng7NN1nDlZ3aciAnW6xbl8eyOU5YVcLCqXsmNQy52SoZNtDZqLmRuWDqNTQcMnXT9Yi/bj9XxRUPAyuLM55kDIqZlTiQc1jla7be66ZmtU2WflFuDpkNKgsK5S1eYnjWRnEnGSTjPlUJXb9haxAajTW1PUOOny/Ip9WTw0q4K9lQ0smHptBhf+RuHa2LsiVInv7HIQC4ZlHgFGAAFkYXM9Yu9FLr7bGVlF4xJMObYNbN5UsXFgDXd/tHiyby6t5LMtMQBva9lEL85uB1J+CIj8MCQszp6NRQBVZeuoCpGPcCuzxrQdB27Kvj/DtXwxuEa/nTWFPJcKWzcf54vfW3s/eISiTaFQrdjQIM0yc1DVnZK4vLG4RpUhZjWo1uP1PC371eSaFN5bv49vHX8Gx4uzCTPlcKHlc0cq/GTZFeYlJxA85UefrZsOpsP1bJuUR5//0EVvSFDi12YnzGkvU1y4+h/1WPeVoTR5ErXdXqjDPpeVwoXv+2yqnBLchxcuNzFvZNTOVrtZ77XSc6kZD442zRuOxLeTGRlp+SaKJ7isILwi++U85Pffs6reyuxKQqqInjA4+SXK0vYU9HIxv1VfNEQsPpQ+zt6CYZ1Xt1bycL8DP7xv87TE9KYlGInpOl8dL4FWZR5a9CivOOlkYHIYLQG1oktskpQBQ3fdhGOPEkByusDuO9K4uNIEP+42k+eK2XQjoRyGMTNQQZySVxMPXzzoVq8rhR2lzdQ6E5j+9Nz2LJqFi++U86nNX5LStmyahbJiTbWL/HGdCbcXe6jszdMSY6Dyx1BMtMSCYb1ISe4S248IU3n+wUuXvqjAgD8HUEyUxOtoA3ww9lT0DEqa+fmpqNhyDBnGtrIz0rly8Z2Xl5ewKYD1Ww/VhfXbiiHQdwcpEYuGZRSTwYL8zPYXe5jbm661eMkupLzIY+Tv1zstbKwtTvLUISg0J3G2YgGLoDq5g4WX6ViUHLjUQUoQhDWdf5hfxV2VeGV5QW8ebiW+tYuq31wgk3hf528iE0VrChx8265j/neDI5Wt+CYYOPcpXZWlLitNZJHiyezYel0q6r34cJMHpvpjlkgX5jvYt+ZS2xbPVtKL6OMzMglMURfCm89UmP9gL9oaLNkluhKzq8utQ/YR29IozLqfh1o7w7JIH4bENZBVQV/cp+b7qBGcbYRiJuv9FpDJeZ7nagCgppOQVYqh6taWDkvh4+rW1hS4KKtK8R8r5N3y338+kgtW1bNsgZNmIEbsDJxIyFwsbu8gWVFWTKI3wBkIJfEYF4Kbz1Sw8b951k5L4cvG9t40DOJTQeqeaQok437z/PLlSU84HHycGGm9YOtuGiMAEtNtBHWdIrcabyyvOBWv6Vxi6lwTUxUY+4Pazq//+oPrF/s5bNvWnltbyX35ThISbTx5LwcPq72U5CVygS7gobRQzxnUgor5+VwrOYyLy8vYP40Fy8vL+CTmssDXrfUk8HrTxRbmfhPflvOu+UNrCjJ5nBVsywOugFIaUUSg5lRPbvjFMuKMnn/TJOVYZ2sK+OMr41tq41Fc9Nm9thMN29+VMvz38vjxNd+WruC5DqTOeNr40rP1cvuJaOL6UTRMapkr/SELcmkKDuNL31thMIazVd6yM9KpepSOzMmp/GzZQWUejLIzUjh42o/21bPoeJiIKYOoL8sUuh2WNv0J1qaW1Hi5h9/dN9V57dKrg9pP5TEZeP+c5ZX3ByYO9QAgWM1LazdWUZXbxhVESjCkFhkq9lbQ0mOg9MXAzGdJed7nZRd+JYn7nfzu9ON1gDlGzWc4VhNi5UQHK5qiRnMPJ6GQYwmN9R+KIRYJoQ4J4SoFkL8zWjsU3LrGKyPxlCDb0s9GTxaPJkEm+FF7grKIH6rUIVhE9R04wce1HSWFLj4uNpPZloi7xyvZ/0Sb4yu/cJCz6haBc2T/rbVs/nHH5XE9FyRwyBGnxEHciGECvwKeAS4F/hzIcS9I92v5NYwVNOjqw2+ff2JYp6bf09MQYnk5hP98c/McbC4wMWJulZUxRjM8XhJNuE47WxG0yooe67cXEYsrQghHgT+b13XH47cfglA1/XXB3uOlFZuX1ZvP8FDXidrFvRlTFuP1LCnopH6y10xl8fP7jjFg55JPLcgz7rvuX86RWdv2Hqu7J9y68jNSKaupZP0ZDuBziBCEfzlIg9vHf+GdYvyCGsMyIxv1vxNyfVxI6WVbKA+6vbFyH39D+B5IcQpIcSp5mZpQ7tdef57eWw+VMtLuyqsLHzzoVrunZzGukV5VkZVcTHAE/e7OVrtZ+3OMrYeqRkQxG2KINGmoMgyzltCXUsnAK2dQTSgODuNDUun80hRJq/urUSN8+sfSj6T3L6MRiCP9zMdkIPpuv6mruuzdV2f7XK5RuFlJTeCiosB1i3KY09FI8/sOMnanWXcl+Ogqa2bzYdqrctsVYFdn/n46cP5gDFcorM3bAXtJQWuqOERclDy7cC5piv85LflvHO8nifn5cSVV47VtLD1yNesKMmOkc9e2lXBS7sqBmwrS+1vD0YjkF8EcqJuTwF8o7BfyS3A7LHygxnfoTuo0R0Mc7Taz8HKZtYt6pNQNh+qZcPSaWw+VMuMrFTrzD11UjKvLC8wNFkh6A3r2BVDt02yy7KFG83EBHXAfQKwq4Ku3jC7y308XuJmebEbiC0AM2WVJ+53s+/MpZgCsD0VjeypaJSl9rcpo6GR24AqYAnQAJwEVuq6fnaw50iN/PbEbFt71hfgtb2VzMlN50RdK2BU+33Z2G5pp6bG+t5pH2d9baiKETBCmqHN1l/uiunbMTFBpaM3LHusjDJpSSpt3eGrbxhhSvoE/Fd6COuw4+k5QF89QMXFgNXxct0iQ2JbmJ/BvjNNMbUDUj+/ddwwjVzX9RDwIvAB8BXwv4YK4pLbl+IpDlZvP8nG/VU8XpJtBXFFwNFqP2lJNjYdrGZhvouN+89z8mu/1U8l0abyoznGhVldSydhTcec2KYIuNIbJjVpYLYouT68rhQE0NYdJteZPCzpyqYI/tDeQ1dQs7aP7oXS2ROy2havWeDhqXlT2V3uY82Ce6yiIKmf356MyrWuruv/qet6vq7rHl3XXx2NfUpuPqWeDOZ7nXQFNd47bahjiTaFJLuK15VCnb8TVcDu8gZKPZM4UNmMIuDJeTnous7bx+tj9hfWjOcD2BWuKXOUDE1Tezc/nptDZloidf5OciYlDzhR5jqTY26nJ9vpDWkk2BQ2LM2PWbhemJ8RE6C3Hqlh65GvY6ymV7OfSm4dskR/nBFv6o9ZaQfwoMfJkfMtBMM607MmUn+5i2BIo6m9h/leJ0er/dgUweGqFstaeMHfSU/UnE3TrdLRGyak6eRlpFidEyWjQ3t3mMPnmunsDVtdCe2qwKYY8pYqjO+l0J3KN5e7uNuZzJmGNubmpvPVpXZqmzusgiBVgXcjZfRvHf8Gf0cP7xyv5+XlBaxZ4OEBj5O1O8sAo11xqSfD6nIo5ZXbA7n6NAa4loq8wYo+Lvg7UBX4xQdV2BTB3Nx0zl26Qk8wzIL8DL6TmsjpiwFWlLgJaTohTbeGJx+t9qPpffYlxwQbwYg+HtZ0GcRHGa8rhbQkFV+gm86eEEerW/C6UgiGdUIazPdmGKX5qqB4yl388czJnGloY77XSXVzB+uXePngbFOMvfTl5QUcrmphYb6Lt4/Xs3JejlVLYFbtPlo8WRb43KbIjHwMYAbn6GKdweYm9h+gvPXI12xYOo1Ct4O1O8tQI1l22TffAobbZLIjiZMRvfxedxp7KhoJhvUBFZzmLX9HkLQklV7ZL2vUEUBKko3l97j57cl6wrpxBRSKnDhLchzMn5ZBgk3wSY2fprZuPqm5zJPzctB0yJmUbC1mmkHY/HfT3hVi08FqVpRkWwOYTczsPRo5VPn2QQbyMUC86fZDXfJGL1qtKMm2FrgeLZ7Mv5ddtAL0ipJs7nWn8nf7zvGzZUbjrFf3VlLkThswODmaJJsi9fBRxuwVruvgTEmwZC1VMab91Pk7me918tZzDwDGyf1kXSv+jt643Sr7N63qr3//cPaUW/E2JdeJlFbGCNfiKIj+0R6uaua+HCMbd01MtDK7nPQk9p25RKHbwT89M5fa5g7++Xg9ExPUIYM4QHcoTqWJZERoOtw3xUFJjoMPK5t5+3g9SwpcFGSlWtt89s23MQ3OtqyaRf3lLj6t8cdcsfVvkhV9Bde/x7zkzkAG8jHCcB0F8ZpinaxrpTsYZtPBajQditxp1Ld2U+qZxIvvlPPrI7Xs+qyBezKSudIrM+0bTXqyPea2ufZQXh/gdH0AIcCZYufDc82c9Rkj1ybYFSY7kmIC8NVO7qYk995pX8xUn8dmuqX+fYchA/kYYKiOhf2J7kpnLoauX+LFrAsTwLmmdp6cl8PBymbunZzKoXPN9IY0DlQ243GlDNinZHRp7QzG9L2I7omiAamJNvwdQQBeWV7AP/6ohA1L8/F92x2jfV/t5G5Kch+cbRo0a5fcGcjBEmOAoSyF5iV0/8df2lVBU1u3tYiZ50oh0Bmkzm80WirJcRDUdM40tGFTDE+4rMq8NaiirzVtgiqsNYwn5+WQMykFVYGN+8+zYek0q6Nh/0k8Q03miTdERHJ7Mlhlp1zsHAPEy5yiHQXxXC17KhoBmJObzsm6VsJhY8HMbH1aXm9kdYowfMnmlBnJzUMAhf0WlqOdQm8fryfXmUydv5NXIp5vME7i/+fuM1ZvHDD+PaxblMebH9UOOOFHZ+0PeJzSiXIHIgP5OCCeq2XLqlmAoYkm2RTO+NoocqdR09xhBQcwFtmUyET1qyF7j48uNlVwxtdGZmoiTe09cbep83eSYFModBvNq8yh2WZDs9rmDh6baTTIMt1J5tVa/xO8LPK5c5HSyhhjKJmlsyc04BJ6+aaPOOtrRxWQnGhj5pS7OFo9MreCOehXcv1MsCsEwzpF2WlMSkngYKXRwz8nPYn61m5rO0UYbRBsqsIPZmTybnmDVZFpzlENhjXsqhJz8jYXM4eS5CS3Hzd0Zqfk9mGwyk1VYcDC19YjNZz1tZNgU0iwKfSGtJgg3r8Pk10V2ONMichMS4y5LYP4yEiyKzzocTIhQeVnywrQdGNRc77XSX1rNznpSYDhbklJtCGEoLM3xO7yBh4vyY6pyPxutoPuoEYwrFkLmuaC6AsLPQMyb7nIeWciA/kYI1pG2bj/nPXDNS+rTVfL2p1l/OKDKl5ZXsCOp+cghLD6pQjRV4CiRP4G0HWdSRMTBrxmU5tx2Z8s+42PCr0hjU9rL7N+iZeKiwF2PD2XvRWNHK32M9+bQX1rtzW4Y05uOsGwRlgzWtQermqOOYl/0RAgya6g6UQ6V2bEDAiRjA3kL28MEu0fXpifQVgjZqGz4mKAR4snM2NyKoVuBxUXA1aXwpQE1ZBGIr1TNB3yMlLwulIIaX1BOx53JQ8M8pLhoyrgdiSh6dDZG+aTGj8vLPSw9UgNn9cb31HOpAnW4I6wptPa0UsorFPoTqOzNxwzDOLFd8rZsmoWf7U0n96Qhk0R7C73xSyCSsYGMpCPQUwnwooSN++W+1AVYixoxVMcvP5EMX+9rIAX3ynn5Nd+WjuDKAJj+INuZON2VbCkwEV1c8ewGl/5At1X3UYyELcjiUSbgq5DU3sPT87LweNK4ZOay2zcf46N+8/zyvICnrg/mz0VjbR3GU1sHsibxNnGdr5f4GLv+gX8cmWJ1Ufl42q/VeSz+VAtK0qyCWlGwN98qFZWbY4xZCAfY0T7hf/xRyW8vLyA1/ZW8pPffs6zO04NsKQ9UpTJgcpmitxpVlGQqXHPn5bBttVzyeqngUtGDwXjBNgb1tB0Y0Byw7dGr/H7p97FpoPVrFlwj+VKCYY1Nh2s5unSXDLTjBPAcwvygD5ZLazBjqfnUurJsGawHq5qZv1iL42B2KIhydhABvIxRnTlJsCaBR4eL8lmd3kDy4oyY7KxYzUt7PrMx3yvkzO+Nubkplv7yXUm83l9gKd+/SmX2npIssl/KqONTRGYXWl03fjML1zuIvuuJF7bW8ln37SyfrGX7cfqWLuzjDxXCnZVIcmu8OaRWvZUNFr9wU36L1aaM1ij10ekRj72kL/OMUZ/J8KxmhYrGztc1WJpqOZC6Ial0/iy0ejXcbKu1eoxbk4DOlrtB2QjrNFEQGQARJ+/Jy3JRp2/k3snp/G7040k2Q1LoUkorLHpQDVbVs3i+QV5lhPlavQ/scs+4mMTGcjHCPGGS2w9UsOzO04NyMYmO5Ks2ZumprqnohFFAUUxdHGA5iu9t+KtjAuiY7AxezNETvoEjla34ExJYNvqOTxdmsumg9X8YMZ3yM9K5dHiyUCfjdSuKtZIvsGQFsPxgQzkY4R4/nGzws/8IVdcDPBIUSaVl9pJUAX/+UUjjxRlsqeiEU3XEQh+OCubj87LhbAbhTD7ikduZ6Ylokfur2/tsqpq91b4YhasHy2ezGMz3THN0basmmVN+pGMb2QgHyPE849vWz3bKg4Bw972zvF6fjxnCol2Y1Dv28frqWxsJznBxl8/Mp3fnW4kFDZ7kk+wnhunDkhyHeh632fpcaXQ2RsmQRXoujHCramthyUFxri1eyencriqhZeXF7D5UK3VblbKJJL+yF4rY4ho//j6xd4Bl9RhDSso/GDGd9hdblyWB8MaT9yfTaHbwdRJyZz1teGYYKO+tct6ruyhMjoIYOYUB2d8bVzwd2JXBX86awp5rhTCmnFl9eZHtdag6/WLvaxZ4LH8/vFkEukJl8hAPoZ4aVcFeyoaYzrZmRrq608UW7rou+UN7C73WbMeNR3+0NbN6u0n0SJDlQNdcuDmSDArYkP9tHBFwJeN7TxaPJnd5T7m3jOJu50pFLpje548vf0khe60AR0J3zhcI/VtyQCktDJGeGlX22Y3tgAAIABJREFUBe+WNwDwgMfJL1eW8OyOk/xb2UX2VDRaOqrZXwUM18SKkmxUAQcqjeERIU0fUJovuT4UIVAVgSoifWpUQVg3Wh38/qs/sH6xl8pL7agK1vqG2ehKEfDN5c4BlZrSNiiJh8zIxxA2VWH9Ei8vvlPOwnwX3UGN7xe4eG5BnnXfu+UNlOQ4qLzUTkjT2VPhw6YqhKNSx3BER3E7kmS15nWi6VgFPgWT0+gOauwubyDXmYwv0M2jxZPZsHS61TrWDNgzIjM4t62eAxD53jLYuP8821bPljKKJC4yIx8jvP5EMVtWzWLzoVq8rhSrE95vVhsVfjOyUq37Hi6azLbVc5h3zySCYZ3JjiQm2JUBGbgv0M3EBPXWvKExQEjT+bKxncrGNt4tb2BFSTaNgW7me528/kQxEFuN+dS8qXxc4+fp0lxL+35q3lR2l/tYs+AeGcQlgzKiQC6E+IUQolIIUSGE2C2EuGu0Dkxy7ZR6MliYn8GJulbm5qZbnfCO1bRQEemC97vTDdRf7uCsL8DH1X6K3GlG8Y8i4i5oymHLI6MnpFFeH+Dl5QUkRU6Wn9Ze5qVdFQAxQx76txke7kBtiWSk0sp/AS/puh4SQvwceAn465EfluR62Hqkht3lRsn9l43trFuUx9qdZYTCGo+XZPPYTDdPbz/J28frAWPmY2OgG3xwpSeMImDqpL7pQJLR40tfO7//qgkw2tQ2tXVbfXHM7ylablm7swzAKsGX03skQzGijFzX9f26rpv2hk+BKSM/JMn1YBYAPTkvhy8b27kvx8GmA9V4XSlWpn3WF2DW3X39VP7lRD0HK5tJtCnWmLY6fydeV8otehd3DsNdB57vdZKgCnaXNzAjKxWbqmBXBcdq/KzdWcbDhZl8UmO0QTBHsoExDPvR4snSMy4ZFqM26k0I8R7wW13X3xrk8eeB5wGmTp0668KFC6PyuhLDknbBb8xmNNvVPvdPp+gJhrGpCtufnsNZX4BX91aSnKDy3Px7+NWH1dZk9vRkOz0hDVWB9u4wExNVrvRISWUk2BRhDeewKYLeyPCHFSVuctKT2XSwmiS7wiNFkweMZ5OZt2QwrnvUmxDi90KIM3H++5OobV4BQsDbg+1H1/U3dV2frev6bJfLdb3vY8wSr1fKsZoW3jhcc9Xti6c4+OBsE2d9AWt7TdcJ60bp96c1fjYdqCY5QUVVBJ9902o8FtlXa2eQUFhj/ZJpADKIjxCBYTf84ewpBMM607NSSU6wUehO491yH1uP1LJ+sZdQWLcWoDcfqmXj/nNWli6DuORauGog13X9B7quF8X57z8AhBB/ATwKPKnfiknOY4TBZm0O5huO3r7Uk8G6RXm8treSc5faWLuzDLuqsH6xFzBGfCXbVX7y36Yxc4qDo9V+cp3J3JfjsJwqvWGd1/+z8qa81zuRa+niqwPpyQmc8bXx5Lwczl26wpZVs3i8xI2qGF7y1Ak27Krx4e8708jC/Aw2HawmGNZiJBaJZDiMSFoRQiwDNgILdV1vHu7zZs+erZ86deq6X3esYgbvp+ZN5a3j31iX128crok77fy90z4+ONtEQVYqXzQErLL7JLtCqccJwMm6VnpCGr0hDbsqCIV17nb2LWg+OS+HS4FuDlQO++sb9wjiD5hWBCya7uKTGj8hzWhCNn+a0xr88OyOUzxxv5vGQDef1Fxm2+rZnPUF+Lt95wiGdRJsCok2ZUCPcYnE5LqllavwSyAV+C8hxOdCiDdGuL9xTXSvlKfmTbV+zINl64/NdPPUvKkcq/HTEwzz+6/+wEMeJ6GwzpGqFj6sbGb9Ei9/en82zhQ7wYie4vu2y8rE/7WsgUPnZBAfDDXOqmZ0EFeEIV/ZFEhJtPHcgjy2rZ7DgmkZ5EyawIeVzfzrqYtW7/f3zzSRmZZkFfcUuh2okS9j9t3pbFk1K+a7lkiGw0hdK15d13N0Xb8v8t8Lo3Vg45HBfMPxOhua8xiNVqfZBMM6obDGrLvTsauCoKazuMDoN15/uRN/R9BomaobMopNEcz3ZtAb0qxFT8lABvts0pPtLC5w8WCeMzIiT1hT7wE+rw/w/zxeZE1n8rpSrEk9rz9RbJ2k3zvts2Swyv+/vfMPjqpM8/3nPd2dnyQxJDGkQzQmIUTDRCPyYxCEFS/igLNgzd51dZwVHURqLXbHGWvuYNXerZpSa9YapoqyLio6sMviXWv2TmZGGZEVBwQz8stIBiRAOoZJ0iEmIdMJCSHdfd77x+lz0t3pJgnkRze8nyqKdM7p7idvJ895zvN+n+c5b7ROUOoUxWhRlZ0xQrBawRwCERyZhUfrYJRvP1ieS5JDY+OKMiRGPnz5rDzuL8uh2nWBkpxUDtZ3Mis/nbbuy9YG54BfcuirTisaVIyccmc6QhjrVtviYcP9JSQn2Ni8t57T57utYR4A+8+0M7cwk8ONXSwuzR6SHvvwZBtvPDE75DMHVGMsxahQjjxGGG4kV3i0bvamfvhOJx+eNApNHDYtoIxo4ZvFWSyflcvhxi5mTkvjREs3dk0YQwwC7+n1S/Sgck6hfPqIuKsgg/VLivi4rh2/LklLtrOyIo9+r5+qGjfLZ+UCWMU+9e29QYqVQRXSe8fdIQoVpRVXXC1jpiMfDWqzc3SEa4vDH2894OLlXXWsqnRaczk37TlDv1dnVaWT3SfayL8pifr2XlZXOnnveGvIvEhNQFF2KvXtvZP4U8Y+mgCbMNJWlQUZfKsijz+6OvlDXTv3BvqHF2al0N3v48HyXIpyUtm8t96a7vP09iP4JWxfM9gQS+nFFaMh2man6n4YB1wpWq9t9nCus5dVlU6qatyW5HDAL/mrshxmTksnJcHGzkNNLC3LIclhaMmDHbkuUU58GLJSHfRc9rOwJItP6zupbe4mJcFObYvHcuILS7I43uxhTmEmH55s464CQzpqFmq9/eQc1u04xmsf11N3vkc5ccWYoSLyOMfsXw2wZkEhr+9vQCJZNCPbkr2t2XaEgsxk/nzhEtlTEnB7+qNK6FRV51BSHBozpqXx4+Vl1DZ7sGnw1idf0dZzGYdNoAnBt76Rx0en2vD5dew2jQduz2X3ifNDWs9u2nPamuD0/LKZk/hTKeKR8ZIfKmIMu01YrWmfe6eGk24PCXYt0Fdc4vb0W6XjiXYNR9hmp3LiQ+nz6pxydwOGFPQX/32Wtp7LFGal4PNLNDE439Rw4jdTVdMypPWs6maoGC9URB7nmMVCn7k6rUgvLdnOpj1nWT4rl9/UuFlVmc8HJ1rp9+rkTEmg/eKAMeDgL5cY8EuSHBr9Xn34N7tOyUxx8Jc+75A7FHvgIicAry5JsBmi8QGfTrJDY1VlPkU5qfw8sB8RvPZrF90WUtQ13D6HQjESVEQe50TrxWISHOmVOzNYu+g2qmrczCnMpKqmBZ9fsrAki/aLAywsyeLPF/osPXm/V6cgM2mif6SrxmETZKY4xuS1NGH0mnHYBAtLskKOTUm087dzpmMPvN+AXzLg0xEYE3xeeaSCcmcGDptRSbutupHNe+t5+8l7hkhIh1MlKRTXgnLkk8hoGmVFq+405z0G68/X7TjGtupGVlc6OdzYZW1uHv7qAo/PK+BgfafV2tanG02dmromb6TbaFWPXr+kq887Ju9pt2kk2IyhGsebPRRmpQCwsCSbyz4/Ow814fXrXLzss55jyjSrXR08vf0oG5aW8M7a+aysyAOwmpcFO+tnFxcPibwXFGcrvbhiTFCOfBIZTaMs0yk8vf0oP3i3xnLefh3WLykaEtndnJbIR6e+5v6yHFISbJTkpDLgl5bm3GRKoo3TgQHAgJE+mGCuJbk32sjcrkGSXUMCqQkaAz6dGblpJCfYmFOYSWfvAJUFGXxa32H1bvfpxsVDArOc6UgJT28/wlsHGnDelMTmvfVUuzp45ZEKNiwt4dXdZzjXaaiAlLNWTATKkU8i0Urvo+VMFxRns3xWLlU1bqtKsGJ6Blv2NVjOv7bZY0WGPr9ObnoSG5aWcEtWCgLouDiAYDCqNDc3/YEU+ZSk+FKkdvV5h8wavRK6NMavFWal0DugY9cEZ7++yIalJeSmG+mlF5aXsarSycH6TsqdaSQFWh9qAjauuJ2NK8rw+iWf1nfy6NwCANbtOGYoUvbWk+jQVAdDxYSiHPkkE61RViSqXR3sP9NBuTONqho3P3j3i5C+K6/vd1nRX6vHSJX86mgzL++q4+O6dquqU2Lkf8MRwIXea0tZTAaRZo2GY95o6BJuDRTtPD6vAE0I7Jrg53vO8H5tK288MRuA/Wc62HB/CQ3tvfT7dGZOS7Mi8Z5LPpITbDxydz5rFxXzxhOz8fp1qw2t6l6omGiUI59kRipJC1Y5vLjiDhLsGlU1LeRlJHHS7bFSMq/vd3GqtdvYlBMCXUordZFgEyQ5NOyaoKffR5Ij9OMP7+oXy9gi/OZGSgvlTEkAIDc9CZsmKMxKobnrEuuXFPHS6gq2PzWHuwpuot+r8418467GXOe0ZDv9XkOh8r8fvoONK8q45DUc9poFhbzySMW4/owKxUhRjnyCiLSxufWAy2qwFKlRVjDhqodEu0aCXaOutZuXd9VZefKmC7180eTBpgkGfP6QaHXAL/H6JcmBnHk0yeGURBu6jNzCNRYQGKkgU7sNhlRw4YxspiTarO9VFmTgl8b/bk8/twUi8R8/NJMt+xqsda4732N1HzR72CwozubT+k42rijj7SfnUNvsodyZQbJD49apKSGT7oMHeThsGut2HFMaccWEEl8J0TjG3NgM1hFv2nOW55fNiChJC781N1MmZmT+xhOzLe24TcDPPjjN/KKpfFrfyePzCnj3SBO+CH7ap0vunJ7B4cYua+ByOP1eP5qI3sJ1stE0kBKaui6RYBtsBPZxXbsxzi5g+9TUBF5YXsaabUcQgKu9l0fnFrB2UTHlzgxrMIf5mZiT6s389vY1c0Pe97l3anj7yTkhOnCzDD942v26Hcd477hbpVcUE4YqCJpAok0AGg1mARAYjuX2aWkcPdfFgE9HYkxtP/zVBQaieGHTeZfkpNLWcxkppbXhqQm4c3oGNU0eUhwafTFaJFSck8pXHb2kJNjovezn3pIsPmu4gE+X5KYl0uf10+/14w1o5w99dQGvX+KwCf7tqbnWmkebvGTKBYOJdu6bnzTwzH1FI3oNheJaiVYQpBz5BDMWvTZM/fLzy2ZQ7szg+/92lL4Bf8QIO/h7dxVkIIEv3d14/ZIXV5Txn4ebcAUaZtkEPDq3gKoaNzclOwJl/QYOzej6Z2LTBpUuV0uqQ6N3lBeLyoIMvmjy4LAJEh22kBmkbk8/Az6jwnJ+cRbfe/uw1RzMYRMkOWxqI1IR16jKzhhgrHpt1DZ7eH7ZDLbsa+Ck28iHJ9o1wq/JaUl2lszMwa4JHIEUxB156SANp97Q3ov7L5eAQTXLzkNNfLNoKp29Azgzkqyhw0ITIfnn0Thxe5Sd09E6cYCT7m5smuAOZzo3pyVysL6T1AQbjZ196LpkdaWTrQe+YletGz1oQVZWONUYNcV1i3LkE8RwE4BGw7OLi1m7qJjXHqtk056zPHD7zcggdYrJt+/M4+BZI1oVCO7IS+fL1m4kki9be6ht9nDJq1sKFV0aTvcPp9vRpaSrb4DkBDurK/MZ8On0XvZHdcrBhP9S+cJuE65GEZNgE2jC2LCdPjWZ400eq+imd8Bs9CX56NTXLCieys5DTdbzTIXPSbdHlcUrrkuUI58ghuu1MdJy/eDzFhRnWz1VBvzSKiF/fF4ByQ6NnYea8EtJfXsvLywv5ZVHKtAIVCr6dE66uxEMpl5MByulUcno0yUrK/L4m3umWxG9ELC0LCfEpnC/bLrtzJTIe+nm+42mKnPRjGx0adjY2NHHzemJ+PTBaF8Txs91y9QUDpztxGETCAEvLJ/J9jVzSEmw8eruM5x0KyeuuP5QjnyCGK7XxkjL9YPPq3Z1sPVAA2BWHZbx4ooy3jnUxOxbp2LTBH5dsroyny37Gti05zT17b1WG1sY/F8I0ITRk0UCUxJseP2SopxU3jvuRghDgy0lHG7sIit10AlLwJmRFPK4MDuFrj6f9T3B0Ei8q88b8jqREMLI3e+ta6cwKwVdQvaUBNq6Lwect2R6ZrIhl9SM1MucwkwWzchmcakxfBrgrb+/h4Uzsti052zEFggKRTyjNjsnkXAlhKlJ/kZ+hjVBprbZM0QtsfWAi1c/PGNtZL7wYCkN7b1WZeKvjjZRVeMmwSaYUziVU+d7AoVD3ayuzOfDk+fpG4jed9zMl1cWZFDb3A1C8uicAj44YUy9OXi2g0SHjZ5+X8TnJ9k1+gPax5QEG0U5qZxo6Y547pU2TZ0ZSdaGq2lTYVYK5zr7uCnFQVefl1nOdE4E7iwkxmDkP1/oA7CqNK9VKaRQxApqszMGCY/CAbx+nWpXp1WuH3zO6/tdbD3gYsu+BmbcPIV+r86Kb0zDr2Npn9860MBHp74mITBgYklZDuuXFPGluxuHTfD+cTf93uhOPNmhWbrssrx0Kqan49fhv461sH5JEX90dTLgl/T0G90Ag4uGNAE2TVhOHIxZoMFOPHjDFAwnromhG6JCwLyiLMsmh12jsiCDc519iEDr2cqCDE66u7FrhhNPsAlc7RfxBV0ZrtQCYTTdJxWKWEY58kkkvGlWcIXg1gNfsfWAK+Scg2fbeWlXHQ/NyqXV08/qSie/qXFj04zX2rC0hD/UtXPL1BQSHTY2rihj8956Nu+tZ+OKMr4zezqSwRy1XRND0h2XvLrlGOtau6n6h4U8Pq+Ayz6dl3bVcSlIafLYvAJsQS+gS/CHbWya0bJDE1QWZHDxsp+lZTmkJdlJCbQIKMpOHbIhKiVU1bSwujKff3yglO1r5vDgrDxWVeYHNmWNvjBJDo3y/AxeXFHGnMKp9Ht17r4lkzeemE1ts+eKSqHRdJ9UKGIZVdk5yQRHjEkOjV8GKgfTku28vKsOgLWLillcmkNVTQuz8tN551ATqyrz2X+mnY0rytiyr4GGgBbcHMJ8b7ERzRblpAJQ7sywqhkvXvZb3Q/tmhhSPOTTjYj8iyYPWw+4KMxODTkugAS7xq+ONuP1S8qd6Zxq7Q7RsAfr1yVwX2k2X3X0sbQshzm3ZTG/2BhibBNGRWaiXUOIwDSewEarAD461cbf3DPdiqTf/KSBDfeX8OaBBs5d6LP0+NWuDrbsa7AcNgytpjUrN83HwRdJlXpRxDPKkY8zw1UPmhHjvcVZ1LYMKirWLjI2QX/2wWn2n27n0/pOVlfms/vEecqd6VTVtLDh/pKQcvP3a1u57NUpd6ZzpPEC1S6jVwjA994+HHCUxqZlY2cf06cm09jZZz3ODOSdYXAT9OXf1w3RpwPckZdGTZOHuwoyKHemc/p8j6XbDh/gbNcEe+vaeXyekWefX5zFln0NvPZYJe8ddzMlyc7KijwevtPJrlo3Ow81UZiVQqunn5KcVJ57p4b1S4rYsq+B9UuKaGjvxRHomrWtupG0ZLv1esEO+8Hy3KhKoeDvmRfSDfeXKCeuiEvGZLNTCPEj4FUgR0o5rDD6RtrsvNKsRiDqMdOhfPetzzhY38nCkiz+4/vzebGqlp2HmlhYksWXrYMbojYNNu+tx69LfH6dgcBQ4ES7hl/CQCBvvToQyT80K9eK7H/7RQt3Ts/g3IVLfHfeLWyrbuRivy/CDEsjpy0ZLOf/84U+Onu9FOek4v7LJZIctpDpPeZFwlTQlDvTafX0Wz9j8IXOrFh95G4nBVNTsWnw8q467irIQAdWVuSxeW89ABuWllh3Ib+pcfP8shnWxc9c95GUyY9F2wSFYqIYtxJ9IUQB8BZQBsxWjnwo0ZzFSKL1p7cfZfatN/FpfSf3lhjpiMfmFVAwNdVKHaxfUmQ14Oq55GPzx4azs2tGekOXhpqjNDfNyjvvP9PO+iVF+HVDOfLSrjoen1fAS6srqHZ1hJS3BxNc1m86aTONYj42sWmCBJtg9q1TOVjfYSlL7i3OYlFpTsgwjGcXF/OTX9cChLSHDVboOGwaKyvyKMpJDYnAr7a3iRqIrIg3xtOR/xfwU+C3wD3KkUdmtD1WfvLrWktOuKA4mx+8+wVVNS0UZqWw74W/Aoy0jSnfq5iewbodx7js09F1iaYJKwqfW5jJqfM9AKxZUMjWA1+FRLCv73fRdKGXd4808+OHDNteCuTng7Fr8O9PzwPgyW1HGPDprK7M573jbsvpawJSE+1sWFrCH12dfFzXHniusM5Jdmg8v6zUiq6DZYKRnKi5dkkOjWcWFY1Z5DyaplkKRSwwLvJDIcS3gRYp5fERnPuMEOKoEOJoe3v7tbxt3HGtPVaqXR18dKqNBJug1dNvPb9iegab9py1hixc9ukM+HTKpqXh1w0nbtcEf2rxWJK8+cVZvP3kPSH9uJ9dXMxLqyu4rzSbl3bV8crvDSe+tCzHUrUY4+EE7x13A0bKJsmh8d7x0J4muoTctEQr7WHi143ugwk2w6H/fM8ZS+HymaszqhMPXjtgRJOURooaiKy4Xhg2IhdCfARMi3DoRWAjsExK6RFCNKIi8iFcy+27WSDk9es4bFrEyHXrARcv76rj1kBawyyQ0QT83dwCdAm//rwFTcDzy0rx61hpmzc/aSAvI4mH73Ratn33rUPo0pis47nkZcAvmeVM58zXF1lYksWRxi6AkH7oYETiCXYNn1/Hpw9O8PHrkJFsx3PJxyxnOucu9JHisNHWc5kFxVncc2smmz+uZ3Wlk5nT0kOcaPh+QvhaqPSH4kbjqiNyKeUDUspZ4f+ABuA24HjAiU8HPhdCRHL6NyzD9Vi5EguKs6nIz6Dfq7NmQWGIZM58/tpFxdxbkk1jZx8Fmcm42nupLMiwVB23ZqWybc0cSqel8UdXJ4BV8HJvSRa//ryFdTuO8dT2w7y6u86SDLZfHGDAL6ksyOD9DYvYvmYOXzR5uDktkTmFxnT5bdWNIX1WHpqVh18fnODj142p855LPgoykzjh7mZuYSZ9Xj9JDo2aP3exrboxRA8fzJufGCoVwBqm8cNlpRQFlCyjvbNRBUCK65WrTq1IKf8kpbxZSlkopSwEmoG7pZTnx8y664BruX2vdnVwKjCGLDglE/z8alcHx851UZCZTFPXJWbfmknVPyzkO7Pz2XmoySoWWlmRxx/q2rFpg/n0zXvr+dGDpfh1ycd17dQ0eXAENihN7nCmW+/52mOVzL1tKkcau1i34xgrK/KsCTmaEFTVtFCen47ESK/YNKMgqCx3Ck1d/cxyprO3rp0NS0v44bJS+r06l71+Pjr1taWHD3a0z9xnSA7N8WsAW/Y18OPlZREvhsM5alUApLheUZWdMcpI2t6a5zy/bAa9A34WlmRzsL6D7771GR+caOPFgHPctOc0W/Y1WM7ys0BkDtBzyReS4/bqEoQxpDnRrvGroy0hF5BXHqlgZUUeADlTEjl34RKPzyvAr0typiRwoqWbx+cV8ItH78KmGYOe69ousrrSiau9N3CuEa2vqsxnwC/5Rn6G1ZY32DmbF48PT7YNyaNHuhgO56jDK2mVQkVxvTBmjjwQmauO/WNEtJTMm58MRq21zR7WLyli8956HizP5T++P4+FJVkcrO9kcWk2axcVh/QZCX68ZkEhaxYUsvnjenQJSQ7N2tgc8Ok8NGsa29bMITGwoRkc2b7ySIX13MWl2Xxwoo1VlU7aLw6wsCSL3x1vZd2OY7zwYCnJCTbuL8th/5kOnl82gw9OtFExPYOK6RnsPnGe1ZX51J3vodrVYfWWCU51XKlXSjgjcdSjeT2FIl5QEXmMEi0l88x9RVbU+eziYksd8vCdTqpdHXzZ2sPqSie7T7Sx9YArRC0T/HhbdSNbDzSQYNcY8On8cFkpS2YafcZtmlFkc9LtCdlgNSPbYCXJ7hNtPDQrl/1nOthwfwlftvYwpzCTlRVGvvyNJ2bzyyfn8tpjlfh1rGpO807C1LM/904NWw+4eHr70ZBcebWrg23VjSwozhqR4mc4R30lBZHKoSviFVWiH2eE9wf58GRbRDXLHU5DzbJxRRlrFxVbvVs2riij3JnBtupG/NIY1vzN4ixL0/3iijIa2ntp6+7n5V11Vk+XaKqb4Nddu6g4qJ9J0ZBI2HwcfLdR7jTSIYtLs62ipi37Gih3GheNdTuOGT9bQH44XDok3FHPL84KaRN8pd4r4b1ZwlUzCkWsovqRxynhBUbhxS3BxULPLi4OeQyMqKrSLEIKLmIa7n1g9EU14T+L6UBvn5ZGbYsnRGp4pdceTuo5kgIgVbKviGXGrbLzalCO/NoIdjbhVZrm8WutTpwohxbtfUZbCQtjV6l5Ne+tUEwE0Ry5Sq2MgFgq5Y6W2gBDUz4W6YDhUhAwNmsS6X3MplkfnGiz0iNpyfaQiD8akY4Hp3RGalO01IxCEauozc4REEv643A1y9pFxWxcUcamPWevWlIXvslnqmFMKWCkIqaxWJNIypxH7nay81AT65cU8fyymaxfUsTLu+qGFAuNByORfCoUsYhKrYyQWMidXikK7rvsu+p0wNW2ERiPNTFz7lv2NViva3ZpHO+7n1i681IoIqFSK9dILAwgiKaqWL+k6JrSAWbE/fT2oywPSAlH0iJ2PNbEfB+zHa85PGMiGIvUjEIxGajUygi51g6GY0Gkghdzcs61pgMWFGezfFYuVTVuFpdmh1wooqVLxmtNYmGtFYp4QkXkI2Akm38TRXgUbBbZXGmc2UiodnWw/0yH1cAKRIh+PNL547EmsbTWCkW8oCLyEXAtHQzHmvBoNTyna9o3mpxusPP8xd9Wsqoyn6qaFhaX5kR1nuO1JrG01gpFvKA2O+OIax1NFm0z781PGnjmvqKQ11xcmsPuE+d5+8l7VCSsUMQI4zIhSDGxXE3mG5oaAAAF5UlEQVS0GiwtNDdLtx5wWd9/7p2aIU7ciMzv4u0n71HyO4UiDlCOPI64mt7mwXrvBcXZli779PkennunhgfLc61zzQsFGBcAldZQKOID5civc8KVLlv2NVg58O/Ou4WH73SGdFOE0E6HaoalQhH7KEd+AxCsdFlcms3+M+3WZimghi0oFHGOcuQ3AKbSxZQWmuXvpgMHuH1a2pAe3qoXt0IRHygd+XVO8AZmbbOHjSvSrX7fZtrlveNuals8JDk0tlU3Mr84Cxjs/a1QKGIb5civc4KVLmakXe7MCCkYCh5OsW7HMZ7afgSHTQvpA65QKGIX5civc4brHxIuaTRncd59S6Zy4gpFnKBy5Dc4wZLG4KpRcyCyQqGIfZQjVwCqF7dCEc8oR64AVI8ThSKeUb1WFAqFIk5QvVYUCoXiOkU5coVCoYhzlCNXKBSKOEc5coVCoYhzlCNXKBSKOGdSVCtCiHbg3IS/8ZXJBuJJNK3sHV+UveOLsvfquFVKmRP+zUlx5LGIEOJoJFlPrKLsHV+UveOLsndsUakVhUKhiHOUI1coFIo4RznyQd6cbANGibJ3fFH2ji/K3jFE5cgVCoUizlERuUKhUMQ5ypErFApFnHPDOnIhxLtCiC8C/xqFEF9EOa9RCPGnwHmT1rJRCPEvQoiWIJu/FeW85UKI00KIeiHE/5poO4PseFUIUSeEqBVCVAkhbopy3qSu73DrJQw2B47XCiHunmgbg2wpEEL8QQhxSghxUgjxjxHOWSKE8AT9nvzzZNgaZM8VP98YW9+ZQev2hRCiWwjxT2HnxNT6Wkgpb/h/wM+Bf45yrBHIjgEb/wX40TDn2AAXUAQkAMeBOybJ3mWAPfD1z4Cfxdr6jmS9gG8BHwACmA8cmsTfgTzg7sDXacCZCPYuAd6fLBtH+/nG0vpG+N04j1GAE7Pra/67YSNyEyGEAP4n8H8n25YxYC5QL6VskFIOAP8J/PVkGCKl3COl9AUefgZMnww7hmEk6/XXwL9Lg8+Am4QQeRNtKICUslVK+Xng6x7gFJA/GbaMITGzvmEsBVxSylirQI/IDe/IgUVAm5TybJTjEtgjhDgmhHhmAu2KxHOB289fCiEyIxzPB5qCHjcTG3/oT2FEXZGYzPUdyXrF5JoKIQqBSuBQhMPfFEIcF0J8IIQon1DDhjLc5xuT6ws8SvTgLpbWFwD7ZBswngghPgKmRTj0opTyt4Gv/44rR+P3SindQoibgf8WQtRJKT8Za1vhyvYCW4CfYvxh/BQjHfRU+EtEeO646UtHsr5CiBcBH7AzystM2PpGYCTrNaFrOhKEEFOA/wf8k5SyO+zw5xjpgIuBfZTfADMm2sYghvt8Y3F9E4BvAz+JcDjW1he4zh25lPKBKx0XQtiBR4DZV3gNd+D/r4UQVRi34+PiaIaz10QIsRV4P8KhZqAg6PF0wD0GpkVkBOv798BKYKkMJBgjvMaErW8ERrJeE7qmwyGEcGA48Z1Syl+HHw927FLK3wsh/o8QIltKOSkNn0bw+cbU+gZ4CPhcStkWfiDW1tfkRk+tPADUSSmbIx0UQqQKIdLMrzE28E5MoH3BtgTnDVdHseMIMEMIcVsgqngU+N1E2BeOEGI58GPg21LKvijnTPb6jmS9fgd8L6CumA94pJStE2ijRWA/523glJRyU5RzpgXOQwgxF+NvvHPirAyxZSSfb8ysbxBR79JjaX2Dua4j8hEwJA8mhHACb0kpvwXkAlWBz80OvCOl3D3hVhr8qxDiLozbzkZgHYTaK6X0CSGeAz7E2HX/pZTy5CTZ+xqQiHE7DfCZlPLZWFrfaOslhHg2cPx14PcYyop6oA9YM1H2ReBe4AngT2JQLrsRuAUse78DrBdC+IBLwKPR7oYmgIifbwyvL0KIFOB/EPj7Cnwv2N5YWl8LVaKvUCgUcc6NnlpRKBSKuEc5coVCoYhzlCNXKBSKOEc5coVCoYhzlCNXKBSKOEc5coVCoYhzlCNXKBSKOOf/A7Oye8ISSiisAAAAAElFTkSuQmCC\n",
- "text/plain": [
- "Classical PCA Theorem
+Classical PCA Theorem
-$$
-J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}}_i)^2,
-$$
-
-
-with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
-\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
-
Proof of the PCA Theorem
+Proof of the PCA Theorem
-$$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{n}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
-$$
-
-
-which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
-
-$$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
-$$
-
-
-Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that
-
-$$
-z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i,
-$$
-
-
-where the vectors on the rhs are known.
+To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
PCA Proof continued
-
-
-$$
-J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2.
-$$
-
-
-
-$$
-\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2,
-$$
-
-
-since the expectation value of
-
-$$
-\mathbb{E}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \mathbb{E}[z_{i0}]= \boldsymbol{w}_0^T\mathbb{E}[\boldsymbol{x}_i]=0,
-$$
-
-
-where we have used the fact that our data are centered.
-
-
-$$
-\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T],
-$$
-
-
-we have thus that
-
-$$
-\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0.
-$$
-
+PCA Proof continued
The final step
+The final step
Geometric Interpretation and link with Singular Value Decomposition
Principal Component Analysis
PCA and scikit-learn
+PCA and scikit-learn
Back to the Cancer Data
+Back to the Cancer Data
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
More on the PCA
+More on the PCA
Incremental PCA
+Incremental PCA
Randomized PCA
+Randomized PCA
Kernel PCA
+Kernel PCA
LLE
+LLE
Other techniques
+Other techniques
Oct 23, 2020
Oct 26, 2020
+Plans for week 43
+
Recurrent Neural Networks
Solving ODEs with Deep Learning
-Ordinary Differential Equations
+Ordinary Differential Equations
-The trial solution
+The trial solution
-Minimization process
+Minimization process
-Minimizing the cost function using gradient descent and automatic differentiation
+Minimizing the cost function using gradient descent and automatic differentiation
-Example: Exponential decay
+Example: Exponential decay
-The function to solve for
+The function to solve for
-The trial solution
+The trial solution
To begin with, a trial solution \( g_t(t) \) must be chosen. A general trial solution for ordinary differential equations could be
$$
@@ -441,7 +444,7 @@ with \( h_1(x) \) ensuring that \( g_t(x) \) satisfies some conditions and \( h_
-Setup of Network
+Setup of Network
-Reformulating the problem
+Reformulating the problem
-More technicalities
+More technicalities
-More details
+More details
-A possible implementation of a neural network
+A possible implementation of a neural network
-Technicalities
+Technicalities
-Final technicalities I
+Final technicalities I
-Final technicalities II
+Final technicalities II
-Final technicalities III
+Final technicalities III
-Final technicalities IV
+Final technicalities IV
-Back propagation
+Back propagation
-Gradient descent
+Gradient descent
-The code for solving the ODE
+The code for solving the ODE
-The network with one input layer, specified number of hidden layers, and one output layer
+The network with one input layer, specified number of hidden layers, and one output layer
-Example: Population growth
+Example: Population growth
-Setting up the problem
+Setting up the problem
-The trial solution
+The trial solution
-The program using Autograd
+The program using Autograd
-Using forward Euler to solve the ODE
+Using forward Euler to solve the ODE
-Example: Solving the one dimensional Poisson equation
+Example: Solving the one dimensional Poisson equation
-The specific equation to solve for
+The specific equation to solve for
-Solving the equation using Autograd
+Solving the equation using Autograd
-Comparing with a numerical scheme
+Comparing with a numerical scheme
-Setting up the code
+Setting up the code
-Partial Differential Equations
+Partial Differential Equations
-Type of problem
+Type of problem
-Network requirements
+Network requirements
-More details
+More details
-Example: The diffusion equation
+Example: The diffusion equation
-Defining the problem
+Defining the problem
-Setting up the network using Autograd
+Setting up the network using Autograd
-Setting up the network using Autograd; The trial solution
+Setting up the network using Autograd; The trial solution
-Why the jacobian?
+Why the jacobian?
-Setting up the network using Autograd; The full program
+Setting up the network using Autograd; The full program
-Example: Solving the wave equation with Neural Networks
+Example: Solving the wave equation with Neural Networks
-The problem to solve for
+The problem to solve for
-The trial solution
+The trial solution
Setting up the network is done in similar matter as for the example of solving the diffusion equation.
The only things we have to change, is the trial solution such that it satisfies the conditions from \eqref{condwave} and the cost function.
@@ -2628,7 +2631,7 @@ Note that this trial solution satisfies the conditions only if \( u(0) = v(0) =
-The analytical solution
+The analytical solution
-Solving the wave equation - the full program using Autograd
+Solving the wave equation - the full program using Autograd
-Resources on differential equations and deep learning
+Resources on differential equations and deep learning
-Friday, Principal Component Analysis
+Friday, Principal Component Analysis
-Basic ideas of the Principal Component Analysis (PCA)
+Basic ideas of the Principal Component Analysis (PCA)
Diagonalize the sample covariance matrix to obtain the principal components
+Diagonalize the sample covariance matrix to obtain the principal components
-Classical PCA Theorem
+Classical PCA Theorem
-Proof of the PCA Theorem
+Proof of the PCA Theorem
-PCA Proof continued
-
-PCA Proof continued
-The final step
+The final step
Geometric Interpretation and link with Singular Value Decomposition
Principal Component Analysis
PCA and scikit-learn
+PCA and scikit-learn
-Back to the Cancer Data
+Back to the Cancer Data
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
-More on the PCA
+More on the PCA
-Incremental PCA
+Incremental PCA
-Randomized PCA
+Randomized PCA
-Kernel PCA
+Kernel PCA
-LLE
+LLE
-Other techniques
+Other techniques
Oct 23, 2020
Oct 26, 2020
+Plans for week 43
+
Recurrent Neural Networks
Solving ODEs with Deep Learning
-Ordinary Differential Equations
+Ordinary Differential Equations
-The trial solution
+The trial solution
-Minimization process
+Minimization process
-Minimizing the cost function using gradient descent and automatic differentiation
+Minimizing the cost function using gradient descent and automatic differentiation
-Example: Exponential decay
+Example: Exponential decay
-The function to solve for
+The function to solve for
-The trial solution
+The trial solution
To begin with, a trial solution \( g_t(t) \) must be chosen. A general trial solution for ordinary differential equations could be
$$
@@ -446,7 +449,7 @@ with \( h_1(x) \) ensuring that \( g_t(x) \) satisfies some conditions and \( h_
-Setup of Network
+Setup of Network
-Reformulating the problem
+Reformulating the problem
-More technicalities
+More technicalities
-More details
+More details
-A possible implementation of a neural network
+A possible implementation of a neural network
-Technicalities
+Technicalities
-Final technicalities I
+Final technicalities I
-Final technicalities II
+Final technicalities II
-Final technicalities III
+Final technicalities III
-Final technicalities IV
+Final technicalities IV
-Back propagation
+Back propagation
-Gradient descent
+Gradient descent
-The code for solving the ODE
+The code for solving the ODE
-The network with one input layer, specified number of hidden layers, and one output layer
+The network with one input layer, specified number of hidden layers, and one output layer
-Example: Population growth
+Example: Population growth
-Setting up the problem
+Setting up the problem
-The trial solution
+The trial solution
-The program using Autograd
+The program using Autograd
-Using forward Euler to solve the ODE
+Using forward Euler to solve the ODE
-Example: Solving the one dimensional Poisson equation
+Example: Solving the one dimensional Poisson equation
-The specific equation to solve for
+The specific equation to solve for
-Solving the equation using Autograd
+Solving the equation using Autograd
-Comparing with a numerical scheme
+Comparing with a numerical scheme
-Setting up the code
+Setting up the code
-Partial Differential Equations
+Partial Differential Equations
-Type of problem
+Type of problem
-Network requirements
+Network requirements
-More details
+More details
-Example: The diffusion equation
+Example: The diffusion equation
-Defining the problem
+Defining the problem
-Setting up the network using Autograd
+Setting up the network using Autograd
-Setting up the network using Autograd; The trial solution
+Setting up the network using Autograd; The trial solution
-Why the jacobian?
+Why the jacobian?
-Setting up the network using Autograd; The full program
+Setting up the network using Autograd; The full program
-Example: Solving the wave equation with Neural Networks
+Example: Solving the wave equation with Neural Networks
-The problem to solve for
+The problem to solve for
-The trial solution
+The trial solution
Setting up the network is done in similar matter as for the example of solving the diffusion equation.
The only things we have to change, is the trial solution such that it satisfies the conditions from \eqref{condwave} and the cost function.
@@ -2633,7 +2636,7 @@ Note that this trial solution satisfies the conditions only if \( u(0) = v(0) =
-The analytical solution
+The analytical solution
-Solving the wave equation - the full program using Autograd
+Solving the wave equation - the full program using Autograd
-Resources on differential equations and deep learning
+Resources on differential equations and deep learning
-Friday, Principal Component Analysis
+Friday, Principal Component Analysis
-Basic ideas of the Principal Component Analysis (PCA)
+Basic ideas of the Principal Component Analysis (PCA)
Diagonalize the sample covariance matrix to obtain the principal components
+Diagonalize the sample covariance matrix to obtain the principal components
-Classical PCA Theorem
+Classical PCA Theorem
-Proof of the PCA Theorem
+Proof of the PCA Theorem
-PCA Proof continued
-
-PCA Proof continued
-The final step
+The final step
Geometric Interpretation and link with Singular Value Decomposition
Principal Component Analysis
PCA and scikit-learn
+PCA and scikit-learn
-Back to the Cancer Data
+Back to the Cancer Data
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
-More on the PCA
+More on the PCA
-Incremental PCA
+Incremental PCA
-Randomized PCA
+Randomized PCA
-Kernel PCA
+Kernel PCA
LLE
-Other techniques
+Other techniques