From f2ee4783c70cf5258ce373c199969160c38fbedd Mon Sep 17 00:00:00 2001 From: mhjensen Date: Thu, 22 Oct 2020 04:33:49 +0200 Subject: [PATCH] test --- doc/pub/week43/html/._week43-bs005.html | 4 +- doc/pub/week43/html/._week43-bs006.html | 4 +- doc/pub/week43/html/._week43-bs010.html | 6 +-- doc/pub/week43/html/._week43-bs011.html | 4 +- doc/pub/week43/html/._week43-bs012.html | 2 +- doc/pub/week43/html/._week43-bs013.html | 16 ++++---- doc/pub/week43/html/week43-reveal.html | 40 +++++++++---------- doc/pub/week43/html/week43-solarized.html | 40 +++++++++---------- doc/pub/week43/html/week43.html | 40 +++++++++---------- doc/pub/week43/ipynb/ipynb-week43-src.tar.gz | Bin 191 -> 191 bytes doc/pub/week43/ipynb/week43.ipynb | 40 +++++++++---------- doc/src/week43/week43.do.txt | 40 +++++++++---------- 12 files changed, 118 insertions(+), 118 deletions(-) diff --git a/doc/pub/week43/html/._week43-bs005.html b/doc/pub/week43/html/._week43-bs005.html index 42f637c4c..c94ebb219 100644 --- a/doc/pub/week43/html/._week43-bs005.html +++ b/doc/pub/week43/html/._week43-bs005.html @@ -365,12 +365,12 @@ c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, , $$

-If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \), +If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \), the cost function becomes $$ \begin{equation} \tag{3} - c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 + c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 \end{equation} $$ diff --git a/doc/pub/week43/html/._week43-bs006.html b/doc/pub/week43/html/._week43-bs006.html index ca319ad6a..7647c10d2 100644 --- a/doc/pub/week43/html/._week43-bs006.html +++ b/doc/pub/week43/html/._week43-bs006.html @@ -326,8 +326,8 @@ MathJax.Hub.Config({

Minimizing the cost function using gradient descent and automatic differentiation

-To perform the minimization using gradient descent, the gradient of \( c\left(\vec x, P\right) \) is needed. -It might happen so that finding an analytical expression of the gradient of \( c(\vec x, P) \) from (3) gets too messy, depending on which cost function one desires to use. +To perform the minimization using gradient descent, the gradient of \( c\left(\boldsymbol{x}, P\right) \) is needed. +It might happen so that finding an analytical expression of the gradient of \( c(\boldsymbol{x}, P) \) from (3) gets too messy, depending on which cost function one desires to use.

Luckily, there exists libraries that makes the job for us through automatic differentiation. diff --git a/doc/pub/week43/html/._week43-bs010.html b/doc/pub/week43/html/._week43-bs010.html index e766a2cc0..998e38d8d 100644 --- a/doc/pub/week43/html/._week43-bs010.html +++ b/doc/pub/week43/html/._week43-bs010.html @@ -389,17 +389,17 @@ $$ $$

-Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes +Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes $$ -\min_{P} c(\vec x, P) +\min_{P} c(\boldsymbol{x}, P) $$

In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could also be expressed as $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\}) $$

diff --git a/doc/pub/week43/html/._week43-bs011.html b/doc/pub/week43/html/._week43-bs011.html index b5fbbed21..6b3b0e94c 100644 --- a/doc/pub/week43/html/._week43-bs011.html +++ b/doc/pub/week43/html/._week43-bs011.html @@ -328,11 +328,11 @@ MathJax.Hub.Config({

A possible implementation of a neural network using Autograd

-For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9). +For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9).

First, the neural network must feed forward the inputs. -This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).

diff --git a/doc/pub/week43/html/._week43-bs012.html b/doc/pub/week43/html/._week43-bs012.html index c8542ee9f..9e08df495 100644 --- a/doc/pub/week43/html/._week43-bs012.html +++ b/doc/pub/week43/html/._week43-bs012.html @@ -332,7 +332,7 @@ The next step is to decide how the parameters should be changed such that they m The chosen cost function for this problem is $$ -c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 $$

diff --git a/doc/pub/week43/html/._week43-bs013.html b/doc/pub/week43/html/._week43-bs013.html index 30f3c897d..8a54161fa 100644 --- a/doc/pub/week43/html/._week43-bs013.html +++ b/doc/pub/week43/html/._week43-bs013.html @@ -329,29 +329,29 @@ MathJax.Hub.Config({ The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.

-In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows: +In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows: $$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega) +\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega}) $$

-for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.

-The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \). +The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$. +The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).

-In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) . +In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .

This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by $$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P) +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P) \end{aligned} $$ diff --git a/doc/pub/week43/html/week43-reveal.html b/doc/pub/week43/html/week43-reveal.html index 9fca70a8f..ad4349ca6 100644 --- a/doc/pub/week43/html/week43-reveal.html +++ b/doc/pub/week43/html/week43-reveal.html @@ -259,13 +259,13 @@ $$

 

-If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \), +If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \), the cost function becomes

 
$$ \begin{equation} \tag{3} - c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 + c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 \end{equation} $$

 
@@ -278,8 +278,8 @@ The neural net should then find some parameters \( P \) that minimizes the cost

Minimizing the cost function using gradient descent and automatic differentiation

-To perform the minimization using gradient descent, the gradient of \( c\left(\vec x, P\right) \) is needed. -It might happen so that finding an analytical expression of the gradient of \( c(\vec x, P) \) from (3) gets too messy, depending on which cost function one desires to use. +To perform the minimization using gradient descent, the gradient of \( c\left(\boldsymbol{x}, P\right) \) is needed. +It might happen so that finding an analytical expression of the gradient of \( c(\boldsymbol{x}, P) \) from (3) gets too messy, depending on which cost function one desires to use.

Luckily, there exists libraries that makes the job for us through automatic differentiation. @@ -461,11 +461,11 @@ $$

 

-Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes +Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes

 
$$ -\min_{P} c(\vec x, P) +\min_{P} c(\boldsymbol{x}, P) $$

 
@@ -474,7 +474,7 @@ In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could al

 
$$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\}) $$

 

@@ -484,11 +484,11 @@ $$

A possible implementation of a neural network using Autograd

-For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9). +For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9).

First, the neural network must feed forward the inputs. -This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).

@@ -611,7 +611,7 @@ The chosen cost function for this problem is

 
$$ -c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 $$

 
@@ -628,23 +628,23 @@ Here, gradient descent with a constant step size has been chosen. The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.

-In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows: +In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows:

 
$$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega) +\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega}) $$

 

-for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.

-The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \). +The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$. +The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).

-In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) . +In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .

This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by @@ -652,8 +652,8 @@ This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated

 
$$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P) +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P) \end{aligned} $$

 
@@ -3444,11 +3444,11 @@ $$

 

-If we let \( \vec x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: +If we let \( \boldsymbol{x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:

 
$$ \begin{equation*} - c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2 + c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\boldsymbol{x) }{\partial x_1}, \dots , \frac{\partial g(\boldsymbol{x) }{\partial x_N}, \frac{\partial g(\boldsymbol{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\boldsymbol{x) }{\partial x_N^n} \right) \right)^2 \end{equation*} $$

 
diff --git a/doc/pub/week43/html/week43-solarized.html b/doc/pub/week43/html/week43-solarized.html index 1b940bbed..680cb7a15 100644 --- a/doc/pub/week43/html/week43-solarized.html +++ b/doc/pub/week43/html/week43-solarized.html @@ -359,12 +359,12 @@ c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, , $$

-If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \), +If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \), the cost function becomes $$ \begin{equation} \label{cost} - c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 + c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 \end{equation} $$ @@ -376,8 +376,8 @@ The neural net should then find some parameters \( P \) that minimizes the cost









Minimizing the cost function using gradient descent and automatic differentiation

-To perform the minimization using gradient descent, the gradient of \( c\left(\vec x, P\right) \) is needed. -It might happen so that finding an analytical expression of the gradient of \( c(\vec x, P) \) from \eqref{cost} gets too messy, depending on which cost function one desires to use. +To perform the minimization using gradient descent, the gradient of \( c\left(\boldsymbol{x}, P\right) \) is needed. +It might happen so that finding an analytical expression of the gradient of \( c(\boldsymbol{x}, P) \) from \eqref{cost} gets too messy, depending on which cost function one desires to use.

Luckily, there exists libraries that makes the job for us through automatic differentiation. @@ -539,17 +539,17 @@ $$ $$

-Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes +Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes $$ -\min_{P} c(\vec x, P) +\min_{P} c(\boldsymbol{x}, P) $$

In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could also be expressed as $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\}) $$

@@ -558,11 +558,11 @@ $$

A possible implementation of a neural network using Autograd

-For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}. +For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.

First, the neural network must feed forward the inputs. -This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).

@@ -672,7 +672,7 @@ The next step is to decide how the parameters should be changed such that they m The chosen cost function for this problem is $$ -c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 $$

@@ -688,29 +688,29 @@ Here, gradient descent with a constant step size has been chosen. The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.

-In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows: +In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows: $$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega) +\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega}) $$

-for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.

-The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \). +The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$. +The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).

-In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) . +In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .

This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by $$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P) +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P) \end{aligned} $$ @@ -3364,10 +3364,10 @@ c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{ $$

-If we let \( \vec x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: +If we let \( \boldsymbol{x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: $$ \begin{equation*} - c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2 + c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\boldsymbol{x) }{\partial x_1}, \dots , \frac{\partial g(\boldsymbol{x) }{\partial x_N}, \frac{\partial g(\boldsymbol{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\boldsymbol{x) }{\partial x_N^n} \right) \right)^2 \end{equation*} $$ diff --git a/doc/pub/week43/html/week43.html b/doc/pub/week43/html/week43.html index 126a31642..f774e68a8 100644 --- a/doc/pub/week43/html/week43.html +++ b/doc/pub/week43/html/week43.html @@ -364,12 +364,12 @@ c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, , $$

-If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \), +If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \), the cost function becomes $$ \begin{equation} \label{cost} - c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 + c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 \end{equation} $$ @@ -381,8 +381,8 @@ The neural net should then find some parameters \( P \) that minimizes the cost









Minimizing the cost function using gradient descent and automatic differentiation

-To perform the minimization using gradient descent, the gradient of \( c\left(\vec x, P\right) \) is needed. -It might happen so that finding an analytical expression of the gradient of \( c(\vec x, P) \) from \eqref{cost} gets too messy, depending on which cost function one desires to use. +To perform the minimization using gradient descent, the gradient of \( c\left(\boldsymbol{x}, P\right) \) is needed. +It might happen so that finding an analytical expression of the gradient of \( c(\boldsymbol{x}, P) \) from \eqref{cost} gets too messy, depending on which cost function one desires to use.

Luckily, there exists libraries that makes the job for us through automatic differentiation. @@ -544,17 +544,17 @@ $$ $$

-Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes +Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes $$ -\min_{P} c(\vec x, P) +\min_{P} c(\boldsymbol{x}, P) $$

In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could also be expressed as $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\}) $$

@@ -563,11 +563,11 @@ $$

A possible implementation of a neural network using Autograd

-For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}. +For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.

First, the neural network must feed forward the inputs. -This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).

@@ -677,7 +677,7 @@ The next step is to decide how the parameters should be changed such that they m The chosen cost function for this problem is $$ -c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 $$

@@ -693,29 +693,29 @@ Here, gradient descent with a constant step size has been chosen. The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.

-In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows: +In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows: $$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega) +\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega}) $$

-for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.

-The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \). +The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$. +The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).

-In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) . +In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .

This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by $$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P) +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P) \end{aligned} $$ @@ -3369,10 +3369,10 @@ c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{ $$

-If we let \( \vec x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: +If we let \( \boldsymbol{x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: $$ \begin{equation*} - c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2 + c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\boldsymbol{x) }{\partial x_1}, \dots , \frac{\partial g(\boldsymbol{x) }{\partial x_N}, \frac{\partial g(\boldsymbol{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\boldsymbol{x) }{\partial x_N^n} \right) \right)^2 \end{equation*} $$ diff --git a/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz b/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz index 583bbaf9251cb7ff6097250dbac5da567c9ba829..edca1817b50c6fe8a1d50b824c954cf4a9f7baf1 100644 GIT binary patch delta 158 zcmV;P0Ac^X0lxtpABzY8i0_bJ00ZsM%?iRW32eK%XS4PWo+*ot8^5T;C*Utn&5CmU) M0FJ{lX8;HQ07eK!0RR91 delta 158 zcmV;P0Ac^X0lxtpABzY8bM25{00ZsM%?iRW3V$_ngV4R=TwMPFd&pMkw!q~JL#o$UfTYY>V(>%emmEXmF0&$(<|`IKXIs(gUzn< zmC>M$$K042ZiqE!iE28XLt)SjEwFfPl|_)c16dTxE2HH(Zmc<4d2vbn>t})>2!gLY M04dY=761qU0PqM&ZvX%Q diff --git a/doc/pub/week43/ipynb/week43.ipynb b/doc/pub/week43/ipynb/week43.ipynb index 6c219bd9d..1550b3bfa 100644 --- a/doc/pub/week43/ipynb/week43.ipynb +++ b/doc/pub/week43/ipynb/week43.ipynb @@ -122,7 +122,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If $N$ inputs are given as a vector $\\vec x$ with elements $x_i$ for $i = 1,\\dots,N$,\n", + "If $N$ inputs are given as a vector $\\boldsymbol{x}$ with elements $x_i$ for $i = 1,\\dots,N$,\n", "the cost function becomes" ] }, @@ -135,7 +135,7 @@ "\n", "$$\n", "\\begin{equation} \\label{cost} \\tag{3}\n", - "\tc\\left(\\vec x, P\\right) = \\frac{1}{N} \\sum_{i=1}^N \\big(f\\left(x_i, \\, g(x_i), \\, g'(x_i), \\, g''(x_i), \\, \\dots \\, , \\, g^{(n)}(x_i)\\right)\\big)^2\n", + "\tc\\left(\\boldsymbol{x}, P\\right) = \\frac{1}{N} \\sum_{i=1}^N \\big(f\\left(x_i, \\, g(x_i), \\, g'(x_i), \\, g''(x_i), \\, \\dots \\, , \\, g^{(n)}(x_i)\\right)\\big)^2\n", "\\end{equation}\n", "$$" ] @@ -148,8 +148,8 @@ "([3](#cost)) for a set of $N$ training samples $x_i$.\n", "\n", "## Minimizing the cost function using gradient descent and automatic differentiation\n", - "To perform the minimization using gradient descent, the gradient of $c\\left(\\vec x, P\\right)$ is needed.\n", - "It might happen so that finding an analytical expression of the gradient of $c(\\vec x, P)$ from ([3](#cost)) gets too messy, depending on which cost function one desires to use.\n", + "To perform the minimization using gradient descent, the gradient of $c\\left(\\boldsymbol{x}, P\\right)$ is needed.\n", + "It might happen so that finding an analytical expression of the gradient of $c(\\boldsymbol{x}, P)$ from ([3](#cost)) gets too messy, depending on which cost function one desires to use.\n", "\n", "Luckily, there exists libraries that makes the job for us through automatic differentiation.\n", "Automatic differentiation is a method of finding the derivatives numerically with very high precision.\n", @@ -391,7 +391,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Letting $\\vec x$ be a vector with elements $x_i$ and $c(\\vec x, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem that our network must solve, becomes" + "Letting $\\boldsymbol{x}$ be a vector with elements $x_i$ and $c(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem that our network must solve, becomes" ] }, { @@ -399,7 +399,7 @@ "metadata": {}, "source": [ "$$\n", - "\\min_{P} c(\\vec x, P)\n", + "\\min_{P} c(\\boldsymbol{x}, P)\n", "$$" ] }, @@ -410,15 +410,15 @@ "In terms of $P_{\\text{hidden} }$ and $P_{\\text{output} }$, this could also be expressed as\n", "\n", "$$\n", - "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} c(\\vec x, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n", + "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} c(\\boldsymbol{x}, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n", "$$\n", "\n", "## A possible implementation of a neural network using Autograd\n", "\n", - "For simplicity, it is assumed that the input is an array $\\vec x = (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills ([9](#min)).\n", + "For simplicity, it is assumed that the input is an array $\\boldsymbol{x} = (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills ([9](#min)).\n", "\n", "First, the neural network must feed forward the inputs.\n", - "This means that $\\vec x$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.\n", + "This means that $\\boldsymbol{x}s$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.\n", "The input layer will consist of $N_{\\text{input} }$ neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be $N_{\\text{hidden} }$.\n", "\n", "For the $i$-th in the hidden layer with weight $w_i^{\\text{hidden} }$ and bias $b_i^{\\text{hidden} }$, the weighting from the $j$-th neuron at the input layer is:" @@ -567,7 +567,7 @@ "metadata": {}, "source": [ "$$\n", - "c(\\vec x, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n", + "c(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n", "$$" ] }, @@ -582,7 +582,7 @@ "## Gradient descent\n", "The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.\n", "\n", - "In general, the update of some parameters $\\vec \\omega$ given a cost function defined by some weights $\\vec \\omega$, $c(\\vec x, \\vec \\omega)$, goes as follows:" + "In general, the update of some parameters $\\boldsymbol{\\omega}$ given a cost function defined by some weights $\\boldsymbol{\\omega}$, $c(\\boldsymbol{x}, \\boldsymbol{\\omega})$, goes as follows:" ] }, { @@ -590,7 +590,7 @@ "metadata": {}, "source": [ "$$\n", - "\\vec \\omega_{\\text{new} } = \\vec \\omega - \\lambda \\nabla_{\\vec \\omega} c(\\vec x, \\vec \\omega)\n", + "\\boldsymbol{\\omega}_{\\text{new} } = \\boldsymbol{\\omega} - \\lambda \\nabla_{\\boldsymbol{\\omega}} c(\\boldsymbol{x, \\boldsymbol{\\omega})\n", "$$" ] }, @@ -598,12 +598,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "for a number of iterations or until $ \\big|\\big| \\vec \\omega_{\\text{new} } - \\vec \\omega \\big|\\big|$ becomes smaller than some given tolerance.\n", + "for a number of iterations or until $ \\big|\\big| \\boldsymbol{\\omega}_{\\text{new} } - \\boldsymbol{\\omega} \\big|\\big|$ becomes smaller than some given tolerance.\n", "\n", - "The value of $\\lambda$ decides how large steps the algorithm must take in the direction of $ \\nabla_{\\vec \\omega} c(\\vec x, \\vec \\omega)$.\n", - "The notation $\\nabla_{\\vec \\omega}$ express the gradient with respect to the elements in $\\vec \\omega$.\n", + "The value of $\\lambda$ decides how large steps the algorithm must take in the direction of $ \\nabla_{\\boldsymbol{\\omega}} c(\\boldsymbol{x}, \\boldsymbol{\\omega})$.\n", + "The notation $\\nabla_{\\boldsymbol{\\omega}}$ express the gradient with respect to the elements in $\\boldsymbol{\\omega$.\n", "\n", - "In our case, we have to minimize the cost function $c(\\vec x, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\\text{hidden} }$ and for the output layer $P_{\\text{output} }$ .\n", + "In our case, we have to minimize the cost function $c(\\boldsymbol{x}, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\\text{hidden} }$ and for the output layer $P_{\\text{output} }$ .\n", "\n", "This means that $P_{\\text{hidden} }$ and $P_{\\text{output} }$ is updated by" ] @@ -614,8 +614,8 @@ "source": [ "$$\n", "\\begin{aligned}\n", - "P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} c(\\vec x, P) \\\\\n", - "P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} c(\\vec x, P)\n", + "P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} c(\\boldsymbol{x}, P) \\\\\n", + "P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} c(\\boldsymbol{x}, P)\n", "\\end{aligned}\n", "$$" ] @@ -3989,7 +3989,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If we let $\\vec x = \\big( x_1, \\dots, x_N \\big)$ be an array containing the values for $x_1, \\dots, x_N$ respectively, the cost function can be reformulated into the following:" + "If we let $\\boldsymbol{x = \\big( x_1, \\dots, x_N \\big)$ be an array containing the values for $x_1, \\dots, x_N$ respectively, the cost function can be reformulated into the following:" ] }, { @@ -3997,7 +3997,7 @@ "metadata": {}, "source": [ "$$\n", - "c\\left(\\vec{x}, P\\right) = f\\left( \\left( \\vec{x}, \\frac{\\partial g(\\vec x) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\vec x) }{\\partial x_N}, \\frac{\\partial g(\\vec x) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\vec x) }{\\partial x_N^n} \\right) \\right)^2\n", + "c\\left(\\vec{x}, P\\right) = f\\left( \\left( \\vec{x}, \\frac{\\partial g(\\boldsymbol{x) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\boldsymbol{x) }{\\partial x_N}, \\frac{\\partial g(\\boldsymbol{x) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\boldsymbol{x) }{\\partial x_N^n} \\right) \\right)^2\n", "$$" ] }, diff --git a/doc/src/week43/week43.do.txt b/doc/src/week43/week43.do.txt index 1e9a47e10..ddf5fa8d4 100644 --- a/doc/src/week43/week43.do.txt +++ b/doc/src/week43/week43.do.txt @@ -75,12 +75,12 @@ The cost function $c\left(x, P \right)$ can therefore be expressed as c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, , \, g^{(n)}(x)\right)\big)^2 !et -If $N$ inputs are given as a vector $\vec x$ with elements $x_i$ for $i = 1,\dots,N$, +If $N$ inputs are given as a vector $\bm{x}$ with elements $x_i$ for $i = 1,\dots,N$, the cost function becomes !bt \begin{equation} \label{cost} - c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 + c\left(\bm{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 \end{equation} !et @@ -89,8 +89,8 @@ The neural net should then find some parameters $P$ that minimizes the cost func !split ===== Minimizing the cost function using gradient descent and automatic differentiation ===== -To perform the minimization using gradient descent, the gradient of $c\left(\vec x, P\right)$ is needed. -It might happen so that finding an analytical expression of the gradient of $c(\vec x, P)$ from (ref{cost}) gets too messy, depending on which cost function one desires to use. +To perform the minimization using gradient descent, the gradient of $c\left(\bm{x}, P\right)$ is needed. +It might happen so that finding an analytical expression of the gradient of $c(\bm{x}, P)$ from (ref{cost}) gets too messy, depending on which cost function one desires to use. Luckily, there exists libraries that makes the job for us through automatic differentiation. Automatic differentiation is a method of finding the derivatives numerically with very high precision. @@ -217,25 +217,25 @@ If the neural network evaluates $g_t(x, P)$ at more values for $x$, say $N$ valu \end{equation} !et -Letting $\vec x$ be a vector with elements $x_i$ and $c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2$ denote the cost function, the minimization problem that our network must solve, becomes +Letting $\bm{x}$ be a vector with elements $x_i$ and $c(\bm{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2$ denote the cost function, the minimization problem that our network must solve, becomes !bt -\min_{P} c(\vec x, P) +\min_{P} c(\bm{x}, P) !et In terms of $P_{\text{hidden} }$ and $P_{\text{output} }$, this could also be expressed as $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\bm{x}, \{P_{\text{hidden} }, P_{\text{output} }\}) $$ !split ===== A possible implementation of a neural network using Autograd ===== -For simplicity, it is assumed that the input is an array $\vec x = (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills (ref{min}). +For simplicity, it is assumed that the input is an array $\bm{x} = (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills (ref{min}). First, the neural network must feed forward the inputs. -This means that $\vec x$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that $\bm{x}s$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of $N_{\text{input} }$ neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be $N_{\text{hidden} }$. For the $i$-th in the hidden layer with weight $w_i^{\text{hidden} }$ and bias $b_i^{\text{hidden} }$, the weighting from the $j$-th neuron at the input layer is: @@ -330,7 +330,7 @@ The next step is to decide how the parameters should be changed such that they m The chosen cost function for this problem is !bt -c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +c(\bm{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 !et In order to minimize the cost function, an optimization method must be chosen. @@ -341,25 +341,25 @@ Here, gradient descent with a constant step size has been chosen. ===== Gradient descent ===== The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum. -In general, the update of some parameters $\vec \omega$ given a cost function defined by some weights $\vec \omega$, $c(\vec x, \vec \omega)$, goes as follows: +In general, the update of some parameters $\bm{\omega}$ given a cost function defined by some weights $\bm{\omega}$, $c(\bm{x}, \bm{\omega})$, goes as follows: !bt -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega) +\bm{\omega}_{\text{new} } = \bm{\omega} - \lambda \nabla_{\bm{\omega}} c(\bm{x, \bm{\omega}) !et -for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \bm{\omega}_{\text{new} } - \bm{\omega} \big|\big|$ becomes smaller than some given tolerance. -The value of $\lambda$ decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation $\nabla_{\vec \omega}$ express the gradient with respect to the elements in $\vec \omega$. +The value of $\lambda$ decides how large steps the algorithm must take in the direction of $ \nabla_{\bm{\omega}} c(\bm{x}, \bm{\omega})$. +The notation $\nabla_{\bm{\omega}}$ express the gradient with respect to the elements in $\bm{\omega$. -In our case, we have to minimize the cost function $c(\vec x, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\text{hidden} }$ and for the output layer $P_{\text{output} }$ . +In our case, we have to minimize the cost function $c(\bm{x}, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\text{hidden} }$ and for the output layer $P_{\text{output} }$ . This means that $P_{\text{hidden} }$ and $P_{\text{output} }$ is updated by !bt \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P) +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\bm{x}, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\bm{x}, P) \end{aligned} !et @@ -2823,10 +2823,10 @@ c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{ \end{equation*} !et -If we let $\vec x = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following: +If we let $\bm{x = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following: !bt \begin{equation*} - c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2 + c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\bm{x) }{\partial x_1}, \dots , \frac{\partial g(\bm{x) }{\partial x_N}, \frac{\partial g(\bm{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\bm{x) }{\partial x_N^n} \right) \right)^2 \end{equation*} !et