lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test

Browse Source
This commit is contained in:
mhjensen
2020-10-22 04:33:49 +02:00
parent 10b2f1e00e
commit f2ee4783c7
12 changed files with 118 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week43/html/._week43-bs005.html
+2 -2
View File
@@ -365,12 +365,12 @@ c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, ,
$$
<p>
If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \),
If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \),
the cost function becomes
$$
\begin{equation} \tag{3}
c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
\end{equation}
$$
doc/pub/week43/html/._week43-bs006.html
+2 -2
View File
@@ -326,8 +326,8 @@ MathJax.Hub.Config({
<!-- !split -->
<h2 id="___sec5" class="anchor">Minimizing the cost function using gradient descent and automatic differentiation </h2>
To perform the minimization using gradient descent, the gradient of \( c\left(\vec x, P\right) \) is needed.
It might happen so that finding an analytical expression of the gradient of \( c(\vec x, P) \) from <a href="._week43-bs005.html#mjx-eqn-3">(3)</a> gets too messy, depending on which cost function one desires to use.
To perform the minimization using gradient descent, the gradient of \( c\left(\boldsymbol{x}, P\right) \) is needed.
It might happen so that finding an analytical expression of the gradient of \( c(\boldsymbol{x}, P) \) from <a href="._week43-bs005.html#mjx-eqn-3">(3)</a> gets too messy, depending on which cost function one desires to use.
<p>
Luckily, there exists libraries that makes the job for us through automatic differentiation.
doc/pub/week43/html/._week43-bs010.html
+3 -3
View File
@@ -389,17 +389,17 @@ $$
$$
<p>
Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
$$
\min_{P} c(\vec x, P)
\min_{P} c(\boldsymbol{x}, P)
$$
<p>
In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could also be expressed as
$$
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\})
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\})
$$
<p>
doc/pub/week43/html/._week43-bs011.html
+2 -2
View File
@@ -328,11 +328,11 @@ MathJax.Hub.Config({
<h2 id="___sec10" class="anchor">A possible implementation of a neural network using Autograd </h2>
<p>
For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills <a href="._week43-bs010.html#mjx-eqn-9">(9)</a>.
For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills <a href="._week43-bs010.html#mjx-eqn-9">(9)</a>.
<p>
First, the neural network must feed forward the inputs.
This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
<p>
doc/pub/week43/html/._week43-bs012.html
+1 -1
View File
@@ -332,7 +332,7 @@ The next step is to decide how the parameters should be changed such that they m
The chosen cost function for this problem is
$$
c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
$$
<p>
doc/pub/week43/html/._week43-bs013.html
+8 -8
View File
@@ -329,29 +329,29 @@ MathJax.Hub.Config({
The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
<p>
In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows:
In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows:
$$
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega)
\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega})
$$
<p>
for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance.
for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.
<p>
The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$.
The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \).
The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$.
The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).
<p>
In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
<p>
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by
$$
\begin{aligned}
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P)
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P)
\end{aligned}
$$
doc/pub/week43/html/week43-reveal.html
+20 -20
View File
@@ -259,13 +259,13 @@ $$
<p>&nbsp;<br>
<p>
If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \),
If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \),
the cost function becomes
<p>&nbsp;<br>
$$
\begin{equation} \tag{3}
c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
\end{equation}
$$
<p>&nbsp;<br>
@@ -278,8 +278,8 @@ The neural net should then find some parameters \( P \) that minimizes the cost
<section>
<h2 id="___sec5">Minimizing the cost function using gradient descent and automatic differentiation </h2>
To perform the minimization using gradient descent, the gradient of \( c\left(\vec x, P\right) \) is needed.
It might happen so that finding an analytical expression of the gradient of \( c(\vec x, P) \) from <a href="#mjx-eqn-3">(3)</a> gets too messy, depending on which cost function one desires to use.
To perform the minimization using gradient descent, the gradient of \( c\left(\boldsymbol{x}, P\right) \) is needed.
It might happen so that finding an analytical expression of the gradient of \( c(\boldsymbol{x}, P) \) from <a href="#mjx-eqn-3">(3)</a> gets too messy, depending on which cost function one desires to use.
<p>
Luckily, there exists libraries that makes the job for us through automatic differentiation.
@@ -461,11 +461,11 @@ $$
<p>&nbsp;<br>
<p>
Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
<p>&nbsp;<br>
$$
\min_{P} c(\vec x, P)
\min_{P} c(\boldsymbol{x}, P)
$$
<p>&nbsp;<br>
@@ -474,7 +474,7 @@ In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could al
<p>&nbsp;<br>
$$
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\})
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\})
$$
<p>&nbsp;<br>
</section>
@@ -484,11 +484,11 @@ $$
<h2 id="___sec10">A possible implementation of a neural network using Autograd </h2>
<p>
For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills <a href="#mjx-eqn-9">(9)</a>.
For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills <a href="#mjx-eqn-9">(9)</a>.
<p>
First, the neural network must feed forward the inputs.
This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
<p>
@@ -611,7 +611,7 @@ The chosen cost function for this problem is
<p>&nbsp;<br>
$$
c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
$$
<p>&nbsp;<br>
@@ -628,23 +628,23 @@ Here, gradient descent with a constant step size has been chosen.
The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
<p>
In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows:
In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows:
<p>&nbsp;<br>
$$
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega)
\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega})
$$
<p>&nbsp;<br>
<p>
for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance.
for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.
<p>
The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$.
The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \).
The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$.
The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).
<p>
In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
<p>
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by
@@ -652,8 +652,8 @@ This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated
<p>&nbsp;<br>
$$
\begin{aligned}
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P)
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P)
\end{aligned}
$$
<p>&nbsp;<br>
@@ -3444,11 +3444,11 @@ $$
<p>&nbsp;<br>
<p>
If we let \( \vec x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
If we let \( \boldsymbol{x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
<p>&nbsp;<br>
$$
\begin{equation*}
c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2
c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\boldsymbol{x) }{\partial x_1}, \dots , \frac{\partial g(\boldsymbol{x) }{\partial x_N}, \frac{\partial g(\boldsymbol{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\boldsymbol{x) }{\partial x_N^n} \right) \right)^2
\end{equation*}
$$
<p>&nbsp;<br>
doc/pub/week43/html/week43-solarized.html
+20 -20
View File
@@ -359,12 +359,12 @@ c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, ,
$$
<p>
If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \),
If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \),
the cost function becomes
$$
\begin{equation} \label{cost}
c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
\end{equation}
$$
@@ -376,8 +376,8 @@ The neural net should then find some parameters \( P \) that minimizes the cost
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Minimizing the cost function using gradient descent and automatic differentiation </h2>
To perform the minimization using gradient descent, the gradient of \( c\left(\vec x, P\right) \) is needed.
It might happen so that finding an analytical expression of the gradient of \( c(\vec x, P) \) from \eqref{cost} gets too messy, depending on which cost function one desires to use.
To perform the minimization using gradient descent, the gradient of \( c\left(\boldsymbol{x}, P\right) \) is needed.
It might happen so that finding an analytical expression of the gradient of \( c(\boldsymbol{x}, P) \) from \eqref{cost} gets too messy, depending on which cost function one desires to use.
<p>
Luckily, there exists libraries that makes the job for us through automatic differentiation.
@@ -539,17 +539,17 @@ $$
$$
<p>
Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
$$
\min_{P} c(\vec x, P)
\min_{P} c(\boldsymbol{x}, P)
$$
<p>
In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could also be expressed as
$$
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\})
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\})
$$
<p>
@@ -558,11 +558,11 @@ $$
<h2 id="___sec10">A possible implementation of a neural network using Autograd </h2>
<p>
For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.
For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.
<p>
First, the neural network must feed forward the inputs.
This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
<p>
@@ -672,7 +672,7 @@ The next step is to decide how the parameters should be changed such that they m
The chosen cost function for this problem is
$$
c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
$$
<p>
@@ -688,29 +688,29 @@ Here, gradient descent with a constant step size has been chosen.
The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
<p>
In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows:
In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows:
$$
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega)
\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega})
$$
<p>
for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance.
for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.
<p>
The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$.
The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \).
The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$.
The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).
<p>
In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
<p>
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by
$$
\begin{aligned}
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P)
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P)
\end{aligned}
$$
@@ -3364,10 +3364,10 @@ c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{
$$
<p>
If we let \( \vec x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
If we let \( \boldsymbol{x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
$$
\begin{equation*}
c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2
c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\boldsymbol{x) }{\partial x_1}, \dots , \frac{\partial g(\boldsymbol{x) }{\partial x_N}, \frac{\partial g(\boldsymbol{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\boldsymbol{x) }{\partial x_N^n} \right) \right)^2
\end{equation*}
$$
doc/pub/week43/html/week43.html
+20 -20
View File
@@ -364,12 +364,12 @@ c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, ,
$$
<p>
If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \),
If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \),
the cost function becomes
$$
\begin{equation} \label{cost}
c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
\end{equation}
$$
@@ -381,8 +381,8 @@ The neural net should then find some parameters \( P \) that minimizes the cost
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Minimizing the cost function using gradient descent and automatic differentiation </h2>
To perform the minimization using gradient descent, the gradient of \( c\left(\vec x, P\right) \) is needed.
It might happen so that finding an analytical expression of the gradient of \( c(\vec x, P) \) from \eqref{cost} gets too messy, depending on which cost function one desires to use.
To perform the minimization using gradient descent, the gradient of \( c\left(\boldsymbol{x}, P\right) \) is needed.
It might happen so that finding an analytical expression of the gradient of \( c(\boldsymbol{x}, P) \) from \eqref{cost} gets too messy, depending on which cost function one desires to use.
<p>
Luckily, there exists libraries that makes the job for us through automatic differentiation.
@@ -544,17 +544,17 @@ $$
$$
<p>
Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
$$
\min_{P} c(\vec x, P)
\min_{P} c(\boldsymbol{x}, P)
$$
<p>
In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could also be expressed as
$$
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\})
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\})
$$
<p>
@@ -563,11 +563,11 @@ $$
<h2 id="___sec10">A possible implementation of a neural network using Autograd </h2>
<p>
For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.
For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.
<p>
First, the neural network must feed forward the inputs.
This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
<p>
@@ -677,7 +677,7 @@ The next step is to decide how the parameters should be changed such that they m
The chosen cost function for this problem is
$$
c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
$$
<p>
@@ -693,29 +693,29 @@ Here, gradient descent with a constant step size has been chosen.
The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
<p>
In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows:
In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows:
$$
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega)
\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega})
$$
<p>
for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance.
for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.
<p>
The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$.
The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \).
The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$.
The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).
<p>
In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
<p>
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by
$$
\begin{aligned}
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P)
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P)
\end{aligned}
$$
@@ -3369,10 +3369,10 @@ c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{
$$
<p>
If we let \( \vec x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
If we let \( \boldsymbol{x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
$$
\begin{equation*}
c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2
c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\boldsymbol{x) }{\partial x_1}, \dots , \frac{\partial g(\boldsymbol{x) }{\partial x_N}, \frac{\partial g(\boldsymbol{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\boldsymbol{x) }{\partial x_N^n} \right) \right)^2
\end{equation*}
$$
doc/pub/week43/ipynb/ipynb-week43-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week43/ipynb/week43.ipynb
+20 -20
View File
@@ -122,7 +122,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If $N$ inputs are given as a vector $\\vec x$ with elements $x_i$ for $i = 1,\\dots,N$,\n",
"If $N$ inputs are given as a vector $\\boldsymbol{x}$ with elements $x_i$ for $i = 1,\\dots,N$,\n",
"the cost function becomes"
]
},
@@ -135,7 +135,7 @@
"\n",
"$$\n",
"\\begin{equation} \\label{cost} \\tag{3}\n",
"\tc\\left(\\vec x, P\\right) = \\frac{1}{N} \\sum_{i=1}^N \\big(f\\left(x_i, \\, g(x_i), \\, g'(x_i), \\, g''(x_i), \\, \\dots \\, , \\, g^{(n)}(x_i)\\right)\\big)^2\n",
"\tc\\left(\\boldsymbol{x}, P\\right) = \\frac{1}{N} \\sum_{i=1}^N \\big(f\\left(x_i, \\, g(x_i), \\, g'(x_i), \\, g''(x_i), \\, \\dots \\, , \\, g^{(n)}(x_i)\\right)\\big)^2\n",
"\\end{equation}\n",
"$$"
]
@@ -148,8 +148,8 @@
"([3](#cost)) for a set of $N$ training samples $x_i$.\n",
"\n",
"## Minimizing the cost function using gradient descent and automatic differentiation\n",
"To perform the minimization using gradient descent, the gradient of $c\\left(\\vec x, P\\right)$ is needed.\n",
"It might happen so that finding an analytical expression of the gradient of $c(\\vec x, P)$ from ([3](#cost)) gets too messy, depending on which cost function one desires to use.\n",
"To perform the minimization using gradient descent, the gradient of $c\\left(\\boldsymbol{x}, P\\right)$ is needed.\n",
"It might happen so that finding an analytical expression of the gradient of $c(\\boldsymbol{x}, P)$ from ([3](#cost)) gets too messy, depending on which cost function one desires to use.\n",
"\n",
"Luckily, there exists libraries that makes the job for us through automatic differentiation.\n",
"Automatic differentiation is a method of finding the derivatives numerically with very high precision.\n",
@@ -391,7 +391,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Letting $\\vec x$ be a vector with elements $x_i$ and $c(\\vec x, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem that our network must solve, becomes"
"Letting $\\boldsymbol{x}$ be a vector with elements $x_i$ and $c(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem that our network must solve, becomes"
]
},
{
@@ -399,7 +399,7 @@
"metadata": {},
"source": [
"$$\n",
"\\min_{P} c(\\vec x, P)\n",
"\\min_{P} c(\\boldsymbol{x}, P)\n",
"$$"
]
},
@@ -410,15 +410,15 @@
"In terms of $P_{\\text{hidden} }$ and $P_{\\text{output} }$, this could also be expressed as\n",
"\n",
"$$\n",
"\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} c(\\vec x, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n",
"\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} c(\\boldsymbol{x}, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n",
"$$\n",
"\n",
"## A possible implementation of a neural network using Autograd\n",
"\n",
"For simplicity, it is assumed that the input is an array $\\vec x = (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills ([9](#min)).\n",
"For simplicity, it is assumed that the input is an array $\\boldsymbol{x} = (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills ([9](#min)).\n",
"\n",
"First, the neural network must feed forward the inputs.\n",
"This means that $\\vec x$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.\n",
"This means that $\\boldsymbol{x}s$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.\n",
"The input layer will consist of $N_{\\text{input} }$ neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be $N_{\\text{hidden} }$.\n",
"\n",
"For the $i$-th in the hidden layer with weight $w_i^{\\text{hidden} }$ and bias $b_i^{\\text{hidden} }$, the weighting from the $j$-th neuron at the input layer is:"
@@ -567,7 +567,7 @@
"metadata": {},
"source": [
"$$\n",
"c(\\vec x, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n",
"c(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n",
"$$"
]
},
@@ -582,7 +582,7 @@
"## Gradient descent\n",
"The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.\n",
"\n",
"In general, the update of some parameters $\\vec \\omega$ given a cost function defined by some weights $\\vec \\omega$, $c(\\vec x, \\vec \\omega)$, goes as follows:"
"In general, the update of some parameters $\\boldsymbol{\\omega}$ given a cost function defined by some weights $\\boldsymbol{\\omega}$, $c(\\boldsymbol{x}, \\boldsymbol{\\omega})$, goes as follows:"
]
},
{
@@ -590,7 +590,7 @@
"metadata": {},
"source": [
"$$\n",
"\\vec \\omega_{\\text{new} } = \\vec \\omega - \\lambda \\nabla_{\\vec \\omega} c(\\vec x, \\vec \\omega)\n",
"\\boldsymbol{\\omega}_{\\text{new} } = \\boldsymbol{\\omega} - \\lambda \\nabla_{\\boldsymbol{\\omega}} c(\\boldsymbol{x, \\boldsymbol{\\omega})\n",
"$$"
]
},
@@ -598,12 +598,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"for a number of iterations or until $ \\big|\\big| \\vec \\omega_{\\text{new} } - \\vec \\omega \\big|\\big|$ becomes smaller than some given tolerance.\n",
"for a number of iterations or until $ \\big|\\big| \\boldsymbol{\\omega}_{\\text{new} } - \\boldsymbol{\\omega} \\big|\\big|$ becomes smaller than some given tolerance.\n",
"\n",
"The value of $\\lambda$ decides how large steps the algorithm must take in the direction of $ \\nabla_{\\vec \\omega} c(\\vec x, \\vec \\omega)$.\n",
"The notation $\\nabla_{\\vec \\omega}$ express the gradient with respect to the elements in $\\vec \\omega$.\n",
"The value of $\\lambda$ decides how large steps the algorithm must take in the direction of $ \\nabla_{\\boldsymbol{\\omega}} c(\\boldsymbol{x}, \\boldsymbol{\\omega})$.\n",
"The notation $\\nabla_{\\boldsymbol{\\omega}}$ express the gradient with respect to the elements in $\\boldsymbol{\\omega$.\n",
"\n",
"In our case, we have to minimize the cost function $c(\\vec x, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\\text{hidden} }$ and for the output layer $P_{\\text{output} }$ .\n",
"In our case, we have to minimize the cost function $c(\\boldsymbol{x}, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\\text{hidden} }$ and for the output layer $P_{\\text{output} }$ .\n",
"\n",
"This means that $P_{\\text{hidden} }$ and $P_{\\text{output} }$ is updated by"
]
@@ -614,8 +614,8 @@
"source": [
"$$\n",
"\\begin{aligned}\n",
"P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} c(\\vec x, P) \\\\\n",
"P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} c(\\vec x, P)\n",
"P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} c(\\boldsymbol{x}, P) \\\\\n",
"P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} c(\\boldsymbol{x}, P)\n",
"\\end{aligned}\n",
"$$"
]
@@ -3989,7 +3989,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"If we let $\\vec x = \\big( x_1, \\dots, x_N \\big)$ be an array containing the values for $x_1, \\dots, x_N$ respectively, the cost function can be reformulated into the following:"
"If we let $\\boldsymbol{x = \\big( x_1, \\dots, x_N \\big)$ be an array containing the values for $x_1, \\dots, x_N$ respectively, the cost function can be reformulated into the following:"
]
},
{
@@ -3997,7 +3997,7 @@
"metadata": {},
"source": [
"$$\n",
"c\\left(\\vec{x}, P\\right) = f\\left( \\left( \\vec{x}, \\frac{\\partial g(\\vec x) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\vec x) }{\\partial x_N}, \\frac{\\partial g(\\vec x) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\vec x) }{\\partial x_N^n} \\right) \\right)^2\n",
"c\\left(\\vec{x}, P\\right) = f\\left( \\left( \\vec{x}, \\frac{\\partial g(\\boldsymbol{x) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\boldsymbol{x) }{\\partial x_N}, \\frac{\\partial g(\\boldsymbol{x) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\boldsymbol{x) }{\\partial x_N^n} \\right) \\right)^2\n",
"$$"
]
},
doc/src/week43/week43.do.txt
+20 -20
View File
@@ -75,12 +75,12 @@ The cost function $c\left(x, P \right)$ can therefore be expressed as
c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, , \, g^{(n)}(x)\right)\big)^2
!et
If $N$ inputs are given as a vector $\vec x$ with elements $x_i$ for $i = 1,\dots,N$,
If $N$ inputs are given as a vector $\bm{x}$ with elements $x_i$ for $i = 1,\dots,N$,
the cost function becomes
!bt
\begin{equation} \label{cost}
c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
c\left(\bm{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
\end{equation}
!et
@@ -89,8 +89,8 @@ The neural net should then find some parameters $P$ that minimizes the cost func
!split
===== Minimizing the cost function using gradient descent and automatic differentiation =====
To perform the minimization using gradient descent, the gradient of $c\left(\vec x, P\right)$ is needed.
It might happen so that finding an analytical expression of the gradient of $c(\vec x, P)$ from (ref{cost}) gets too messy, depending on which cost function one desires to use.
To perform the minimization using gradient descent, the gradient of $c\left(\bm{x}, P\right)$ is needed.
It might happen so that finding an analytical expression of the gradient of $c(\bm{x}, P)$ from (ref{cost}) gets too messy, depending on which cost function one desires to use.
Luckily, there exists libraries that makes the job for us through automatic differentiation.
Automatic differentiation is a method of finding the derivatives numerically with very high precision.
@@ -217,25 +217,25 @@ If the neural network evaluates $g_t(x, P)$ at more values for $x$, say $N$ valu
\end{equation}
!et
Letting $\vec x$ be a vector with elements $x_i$ and $c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2$ denote the cost function, the minimization problem that our network must solve, becomes
Letting $\bm{x}$ be a vector with elements $x_i$ and $c(\bm{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2$ denote the cost function, the minimization problem that our network must solve, becomes
!bt
\min_{P} c(\vec x, P)
\min_{P} c(\bm{x}, P)
!et
In terms of $P_{\text{hidden} }$ and $P_{\text{output} }$, this could also be expressed as
$$
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\})
\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\bm{x}, \{P_{\text{hidden} }, P_{\text{output} }\})
$$
!split
===== A possible implementation of a neural network using Autograd =====
For simplicity, it is assumed that the input is an array $\vec x = (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills (ref{min}).
For simplicity, it is assumed that the input is an array $\bm{x} = (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills (ref{min}).
First, the neural network must feed forward the inputs.
This means that $\vec x$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
This means that $\bm{x}s$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
The input layer will consist of $N_{\text{input} }$ neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be $N_{\text{hidden} }$.
For the $i$-th in the hidden layer with weight $w_i^{\text{hidden} }$ and bias $b_i^{\text{hidden} }$, the weighting from the $j$-th neuron at the input layer is:
@@ -330,7 +330,7 @@ The next step is to decide how the parameters should be changed such that they m
The chosen cost function for this problem is
!bt
c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
c(\bm{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
!et
In order to minimize the cost function, an optimization method must be chosen.
@@ -341,25 +341,25 @@ Here, gradient descent with a constant step size has been chosen.
===== Gradient descent =====
The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
In general, the update of some parameters $\vec \omega$ given a cost function defined by some weights $\vec \omega$, $c(\vec x, \vec \omega)$, goes as follows:
In general, the update of some parameters $\bm{\omega}$ given a cost function defined by some weights $\bm{\omega}$, $c(\bm{x}, \bm{\omega})$, goes as follows:
!bt
\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega)
\bm{\omega}_{\text{new} } = \bm{\omega} - \lambda \nabla_{\bm{\omega}} c(\bm{x, \bm{\omega})
!et
for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance.
for a number of iterations or until $ \big|\big| \bm{\omega}_{\text{new} } - \bm{\omega} \big|\big|$ becomes smaller than some given tolerance.
The value of $\lambda$ decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$.
The notation $\nabla_{\vec \omega}$ express the gradient with respect to the elements in $\vec \omega$.
The value of $\lambda$ decides how large steps the algorithm must take in the direction of $ \nabla_{\bm{\omega}} c(\bm{x}, \bm{\omega})$.
The notation $\nabla_{\bm{\omega}}$ express the gradient with respect to the elements in $\bm{\omega$.
In our case, we have to minimize the cost function $c(\vec x, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\text{hidden} }$ and for the output layer $P_{\text{output} }$ .
In our case, we have to minimize the cost function $c(\bm{x}, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\text{hidden} }$ and for the output layer $P_{\text{output} }$ .
This means that $P_{\text{hidden} }$ and $P_{\text{output} }$ is updated by
!bt
\begin{aligned}
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P)
P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\bm{x}, P) \\
P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\bm{x}, P)
\end{aligned}
!et
@@ -2823,10 +2823,10 @@ c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{
\end{equation*}
!et
If we let $\vec x = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following:
If we let $\bm{x = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following:
!bt
\begin{equation*}
c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2
c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\bm{x) }{\partial x_1}, \dots , \frac{\partial g(\bm{x) }{\partial x_N}, \frac{\partial g(\bm{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\bm{x) }{\partial x_N^n} \right) \right)^2
\end{equation*}
!et