-If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \), +If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \), the cost function becomes $$ \begin{equation} \tag{3} - c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 + c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 \end{equation} $$ diff --git a/doc/pub/week43/html/._week43-bs006.html b/doc/pub/week43/html/._week43-bs006.html index ca319ad6a..7647c10d2 100644 --- a/doc/pub/week43/html/._week43-bs006.html +++ b/doc/pub/week43/html/._week43-bs006.html @@ -326,8 +326,8 @@ MathJax.Hub.Config({
Luckily, there exists libraries that makes the job for us through automatic differentiation. diff --git a/doc/pub/week43/html/._week43-bs010.html b/doc/pub/week43/html/._week43-bs010.html index e766a2cc0..998e38d8d 100644 --- a/doc/pub/week43/html/._week43-bs010.html +++ b/doc/pub/week43/html/._week43-bs010.html @@ -389,17 +389,17 @@ $$ $$
-Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes +Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes $$ -\min_{P} c(\vec x, P) +\min_{P} c(\boldsymbol{x}, P) $$
In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could also be expressed as $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\}) $$
diff --git a/doc/pub/week43/html/._week43-bs011.html b/doc/pub/week43/html/._week43-bs011.html index b5fbbed21..6b3b0e94c 100644 --- a/doc/pub/week43/html/._week43-bs011.html +++ b/doc/pub/week43/html/._week43-bs011.html @@ -328,11 +328,11 @@ MathJax.Hub.Config({
-For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9). +For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9).
First, the neural network must feed forward the inputs. -This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
diff --git a/doc/pub/week43/html/._week43-bs012.html b/doc/pub/week43/html/._week43-bs012.html index c8542ee9f..9e08df495 100644 --- a/doc/pub/week43/html/._week43-bs012.html +++ b/doc/pub/week43/html/._week43-bs012.html @@ -332,7 +332,7 @@ The next step is to decide how the parameters should be changed such that they m The chosen cost function for this problem is $$ -c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 $$
diff --git a/doc/pub/week43/html/._week43-bs013.html b/doc/pub/week43/html/._week43-bs013.html index 30f3c897d..8a54161fa 100644 --- a/doc/pub/week43/html/._week43-bs013.html +++ b/doc/pub/week43/html/._week43-bs013.html @@ -329,29 +329,29 @@ MathJax.Hub.Config({ The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
-In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows: +In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows: $$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega) +\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega}) $$
-for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.
-The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \). +The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$. +The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).
-In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) . +In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by $$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P) +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P) \end{aligned} $$ diff --git a/doc/pub/week43/html/week43-reveal.html b/doc/pub/week43/html/week43-reveal.html index 9fca70a8f..ad4349ca6 100644 --- a/doc/pub/week43/html/week43-reveal.html +++ b/doc/pub/week43/html/week43-reveal.html @@ -259,13 +259,13 @@ $$
-If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \), +If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \), the cost function becomes
$$
\begin{equation} \tag{3}
- c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
+ c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
\end{equation}
$$
Luckily, there exists libraries that makes the job for us through automatic differentiation.
@@ -461,11 +461,11 @@ $$
-Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
+Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes
@@ -278,8 +278,8 @@ The neural net should then find some parameters \( P \) that minimizes the cost
Minimizing the cost function using gradient descent and automatic differentiation
-To perform the minimization using gradient descent, the gradient of \( c\left(\vec x, P\right) \) is needed.
-It might happen so that finding an analytical expression of the gradient of \( c(\vec x, P) \) from (3) gets too messy, depending on which cost function one desires to use.
+To perform the minimization using gradient descent, the gradient of \( c\left(\boldsymbol{x}, P\right) \) is needed.
+It might happen so that finding an analytical expression of the gradient of \( c(\boldsymbol{x}, P) \) from (3) gets too messy, depending on which cost function one desires to use.
$$
-\min_{P} c(\vec x, P)
+\min_{P} c(\boldsymbol{x}, P)
$$
@@ -474,7 +474,7 @@ In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could al
$$
-\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\})
+\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\})
$$
-For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9). +For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills (9).
First, the neural network must feed forward the inputs. -This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
@@ -611,7 +611,7 @@ The chosen cost function for this problem is
$$
-c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
+c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
$$
@@ -628,23 +628,23 @@ Here, gradient descent with a constant step size has been chosen.
The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
-In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows: +In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows:
$$
-\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega)
+\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega})
$$
-for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.
-The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \). +The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$. +The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).
-In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) . +In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by @@ -652,8 +652,8 @@ This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated
$$
\begin{aligned}
-P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\
-P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P)
+P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\
+P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P)
\end{aligned}
$$
@@ -3444,11 +3444,11 @@ $$
-If we let \( \vec x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: +If we let \( \boldsymbol{x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following:
$$
\begin{equation*}
- c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2
+ c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\boldsymbol{x) }{\partial x_1}, \dots , \frac{\partial g(\boldsymbol{x) }{\partial x_N}, \frac{\partial g(\boldsymbol{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\boldsymbol{x) }{\partial x_N^n} \right) \right)^2
\end{equation*}
$$
diff --git a/doc/pub/week43/html/week43-solarized.html b/doc/pub/week43/html/week43-solarized.html
index 1b940bbed..680cb7a15 100644
--- a/doc/pub/week43/html/week43-solarized.html
+++ b/doc/pub/week43/html/week43-solarized.html
@@ -359,12 +359,12 @@ c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, ,
$$
-If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \),
+If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \),
the cost function becomes
$$
\begin{equation} \label{cost}
- c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
+ c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
\end{equation}
$$
@@ -376,8 +376,8 @@ The neural net should then find some parameters \( P \) that minimizes the cost
Luckily, there exists libraries that makes the job for us through automatic differentiation. @@ -539,17 +539,17 @@ $$ $$
-Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes +Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes $$ -\min_{P} c(\vec x, P) +\min_{P} c(\boldsymbol{x}, P) $$
In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could also be expressed as $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\}) $$
@@ -558,11 +558,11 @@ $$
-For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}. +For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.
First, the neural network must feed forward the inputs. -This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
@@ -672,7 +672,7 @@ The next step is to decide how the parameters should be changed such that they m The chosen cost function for this problem is $$ -c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 $$
@@ -688,29 +688,29 @@ Here, gradient descent with a constant step size has been chosen. The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
-In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows: +In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows: $$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega) +\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega}) $$
-for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.
-The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \). +The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$. +The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).
-In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) . +In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by $$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P) +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P) \end{aligned} $$ @@ -3364,10 +3364,10 @@ c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{ $$
-If we let \( \vec x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: +If we let \( \boldsymbol{x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: $$ \begin{equation*} - c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2 + c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\boldsymbol{x) }{\partial x_1}, \dots , \frac{\partial g(\boldsymbol{x) }{\partial x_N}, \frac{\partial g(\boldsymbol{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\boldsymbol{x) }{\partial x_N^n} \right) \right)^2 \end{equation*} $$ diff --git a/doc/pub/week43/html/week43.html b/doc/pub/week43/html/week43.html index 126a31642..f774e68a8 100644 --- a/doc/pub/week43/html/week43.html +++ b/doc/pub/week43/html/week43.html @@ -364,12 +364,12 @@ c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, , $$
-If \( N \) inputs are given as a vector \( \vec x \) with elements \( x_i \) for \( i = 1,\dots,N \),
+If \( N \) inputs are given as a vector \( \boldsymbol{x} \) with elements \( x_i \) for \( i = 1,\dots,N \),
the cost function becomes
$$
\begin{equation} \label{cost}
- c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
+ c\left(\boldsymbol{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
\end{equation}
$$
@@ -381,8 +381,8 @@ The neural net should then find some parameters \( P \) that minimizes the cost
Luckily, there exists libraries that makes the job for us through automatic differentiation. @@ -544,17 +544,17 @@ $$ $$
-Letting \( \vec x \) be a vector with elements \( x_i \) and \( c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes +Letting \( \boldsymbol{x} \) be a vector with elements \( x_i \) and \( c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 \) denote the cost function, the minimization problem that our network must solve, becomes $$ -\min_{P} c(\vec x, P) +\min_{P} c(\boldsymbol{x}, P) $$
In terms of \( P_{\text{hidden} } \) and \( P_{\text{output} } \), this could also be expressed as $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\boldsymbol{x}, \{P_{\text{hidden} }, P_{\text{output} }\}) $$
@@ -563,11 +563,11 @@ $$
-For simplicity, it is assumed that the input is an array \( \vec x = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}. +For simplicity, it is assumed that the input is an array \( \boldsymbol{x} = (x_1, \dots, x_N) \) with \( N \) elements. It is at these points the neural network should find \( P \) such that it fulfills \eqref{min}.
First, the neural network must feed forward the inputs. -This means that \( \vec x \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that \( \boldsymbol{x}s \) must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of \( N_{\text{input} } \) neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be \( N_{\text{hidden} } \).
@@ -677,7 +677,7 @@ The next step is to decide how the parameters should be changed such that they m The chosen cost function for this problem is $$ -c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +c(\boldsymbol{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 $$
@@ -693,29 +693,29 @@ Here, gradient descent with a constant step size has been chosen. The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
-In general, the update of some parameters \( \vec \omega \) given a cost function defined by some weights \( \vec \omega \), \( c(\vec x, \vec \omega) \), goes as follows: +In general, the update of some parameters \( \boldsymbol{\omega} \) given a cost function defined by some weights \( \boldsymbol{\omega} \), \( c(\boldsymbol{x}, \boldsymbol{\omega}) \), goes as follows: $$ -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega) +\boldsymbol{\omega}_{\text{new} } = \boldsymbol{\omega} - \lambda \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x, \boldsymbol{\omega}) $$
-for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \boldsymbol{\omega}_{\text{new} } - \boldsymbol{\omega} \big|\big|$ becomes smaller than some given tolerance.
-The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation \( \nabla_{\vec \omega} \) express the gradient with respect to the elements in \( \vec \omega \). +The value of \( \lambda \) decides how large steps the algorithm must take in the direction of $ \nabla_{\boldsymbol{\omega}} c(\boldsymbol{x}, \boldsymbol{\omega})$. +The notation \( \nabla_{\boldsymbol{\omega}} \) express the gradient with respect to the elements in \( \boldsymbol{\omega \).
-In our case, we have to minimize the cost function \( c(\vec x, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) . +In our case, we have to minimize the cost function \( c(\boldsymbol{x}, P) \) with respect to the two sets of weights and biases, that is for the hidden layer \( P_{\text{hidden} } \) and for the output layer \( P_{\text{output} } \) .
This means that \( P_{\text{hidden} } \) and \( P_{\text{output} } \) is updated by $$ \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P) +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\boldsymbol{x}, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\boldsymbol{x}, P) \end{aligned} $$ @@ -3369,10 +3369,10 @@ c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{ $$
-If we let \( \vec x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: +If we let \( \boldsymbol{x = \big( x_1, \dots, x_N \big) \) be an array containing the values for \( x_1, \dots, x_N \) respectively, the cost function can be reformulated into the following: $$ \begin{equation*} - c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2 + c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\boldsymbol{x) }{\partial x_1}, \dots , \frac{\partial g(\boldsymbol{x) }{\partial x_N}, \frac{\partial g(\boldsymbol{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\boldsymbol{x) }{\partial x_N^n} \right) \right)^2 \end{equation*} $$ diff --git a/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz b/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz index 583bbaf92..edca1817b 100644 Binary files a/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz and b/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz differ diff --git a/doc/pub/week43/ipynb/week43.ipynb b/doc/pub/week43/ipynb/week43.ipynb index 6c219bd9d..1550b3bfa 100644 --- a/doc/pub/week43/ipynb/week43.ipynb +++ b/doc/pub/week43/ipynb/week43.ipynb @@ -122,7 +122,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If $N$ inputs are given as a vector $\\vec x$ with elements $x_i$ for $i = 1,\\dots,N$,\n", + "If $N$ inputs are given as a vector $\\boldsymbol{x}$ with elements $x_i$ for $i = 1,\\dots,N$,\n", "the cost function becomes" ] }, @@ -135,7 +135,7 @@ "\n", "$$\n", "\\begin{equation} \\label{cost} \\tag{3}\n", - "\tc\\left(\\vec x, P\\right) = \\frac{1}{N} \\sum_{i=1}^N \\big(f\\left(x_i, \\, g(x_i), \\, g'(x_i), \\, g''(x_i), \\, \\dots \\, , \\, g^{(n)}(x_i)\\right)\\big)^2\n", + "\tc\\left(\\boldsymbol{x}, P\\right) = \\frac{1}{N} \\sum_{i=1}^N \\big(f\\left(x_i, \\, g(x_i), \\, g'(x_i), \\, g''(x_i), \\, \\dots \\, , \\, g^{(n)}(x_i)\\right)\\big)^2\n", "\\end{equation}\n", "$$" ] @@ -148,8 +148,8 @@ "([3](#cost)) for a set of $N$ training samples $x_i$.\n", "\n", "## Minimizing the cost function using gradient descent and automatic differentiation\n", - "To perform the minimization using gradient descent, the gradient of $c\\left(\\vec x, P\\right)$ is needed.\n", - "It might happen so that finding an analytical expression of the gradient of $c(\\vec x, P)$ from ([3](#cost)) gets too messy, depending on which cost function one desires to use.\n", + "To perform the minimization using gradient descent, the gradient of $c\\left(\\boldsymbol{x}, P\\right)$ is needed.\n", + "It might happen so that finding an analytical expression of the gradient of $c(\\boldsymbol{x}, P)$ from ([3](#cost)) gets too messy, depending on which cost function one desires to use.\n", "\n", "Luckily, there exists libraries that makes the job for us through automatic differentiation.\n", "Automatic differentiation is a method of finding the derivatives numerically with very high precision.\n", @@ -391,7 +391,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Letting $\\vec x$ be a vector with elements $x_i$ and $c(\\vec x, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem that our network must solve, becomes" + "Letting $\\boldsymbol{x}$ be a vector with elements $x_i$ and $c(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem that our network must solve, becomes" ] }, { @@ -399,7 +399,7 @@ "metadata": {}, "source": [ "$$\n", - "\\min_{P} c(\\vec x, P)\n", + "\\min_{P} c(\\boldsymbol{x}, P)\n", "$$" ] }, @@ -410,15 +410,15 @@ "In terms of $P_{\\text{hidden} }$ and $P_{\\text{output} }$, this could also be expressed as\n", "\n", "$$\n", - "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} c(\\vec x, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n", + "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} c(\\boldsymbol{x}, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n", "$$\n", "\n", "## A possible implementation of a neural network using Autograd\n", "\n", - "For simplicity, it is assumed that the input is an array $\\vec x = (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills ([9](#min)).\n", + "For simplicity, it is assumed that the input is an array $\\boldsymbol{x} = (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills ([9](#min)).\n", "\n", "First, the neural network must feed forward the inputs.\n", - "This means that $\\vec x$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.\n", + "This means that $\\boldsymbol{x}s$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.\n", "The input layer will consist of $N_{\\text{input} }$ neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be $N_{\\text{hidden} }$.\n", "\n", "For the $i$-th in the hidden layer with weight $w_i^{\\text{hidden} }$ and bias $b_i^{\\text{hidden} }$, the weighting from the $j$-th neuron at the input layer is:" @@ -567,7 +567,7 @@ "metadata": {}, "source": [ "$$\n", - "c(\\vec x, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n", + "c(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n", "$$" ] }, @@ -582,7 +582,7 @@ "## Gradient descent\n", "The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.\n", "\n", - "In general, the update of some parameters $\\vec \\omega$ given a cost function defined by some weights $\\vec \\omega$, $c(\\vec x, \\vec \\omega)$, goes as follows:" + "In general, the update of some parameters $\\boldsymbol{\\omega}$ given a cost function defined by some weights $\\boldsymbol{\\omega}$, $c(\\boldsymbol{x}, \\boldsymbol{\\omega})$, goes as follows:" ] }, { @@ -590,7 +590,7 @@ "metadata": {}, "source": [ "$$\n", - "\\vec \\omega_{\\text{new} } = \\vec \\omega - \\lambda \\nabla_{\\vec \\omega} c(\\vec x, \\vec \\omega)\n", + "\\boldsymbol{\\omega}_{\\text{new} } = \\boldsymbol{\\omega} - \\lambda \\nabla_{\\boldsymbol{\\omega}} c(\\boldsymbol{x, \\boldsymbol{\\omega})\n", "$$" ] }, @@ -598,12 +598,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "for a number of iterations or until $ \\big|\\big| \\vec \\omega_{\\text{new} } - \\vec \\omega \\big|\\big|$ becomes smaller than some given tolerance.\n", + "for a number of iterations or until $ \\big|\\big| \\boldsymbol{\\omega}_{\\text{new} } - \\boldsymbol{\\omega} \\big|\\big|$ becomes smaller than some given tolerance.\n", "\n", - "The value of $\\lambda$ decides how large steps the algorithm must take in the direction of $ \\nabla_{\\vec \\omega} c(\\vec x, \\vec \\omega)$.\n", - "The notation $\\nabla_{\\vec \\omega}$ express the gradient with respect to the elements in $\\vec \\omega$.\n", + "The value of $\\lambda$ decides how large steps the algorithm must take in the direction of $ \\nabla_{\\boldsymbol{\\omega}} c(\\boldsymbol{x}, \\boldsymbol{\\omega})$.\n", + "The notation $\\nabla_{\\boldsymbol{\\omega}}$ express the gradient with respect to the elements in $\\boldsymbol{\\omega$.\n", "\n", - "In our case, we have to minimize the cost function $c(\\vec x, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\\text{hidden} }$ and for the output layer $P_{\\text{output} }$ .\n", + "In our case, we have to minimize the cost function $c(\\boldsymbol{x}, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\\text{hidden} }$ and for the output layer $P_{\\text{output} }$ .\n", "\n", "This means that $P_{\\text{hidden} }$ and $P_{\\text{output} }$ is updated by" ] @@ -614,8 +614,8 @@ "source": [ "$$\n", "\\begin{aligned}\n", - "P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} c(\\vec x, P) \\\\\n", - "P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} c(\\vec x, P)\n", + "P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} c(\\boldsymbol{x}, P) \\\\\n", + "P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} c(\\boldsymbol{x}, P)\n", "\\end{aligned}\n", "$$" ] @@ -3989,7 +3989,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If we let $\\vec x = \\big( x_1, \\dots, x_N \\big)$ be an array containing the values for $x_1, \\dots, x_N$ respectively, the cost function can be reformulated into the following:" + "If we let $\\boldsymbol{x = \\big( x_1, \\dots, x_N \\big)$ be an array containing the values for $x_1, \\dots, x_N$ respectively, the cost function can be reformulated into the following:" ] }, { @@ -3997,7 +3997,7 @@ "metadata": {}, "source": [ "$$\n", - "c\\left(\\vec{x}, P\\right) = f\\left( \\left( \\vec{x}, \\frac{\\partial g(\\vec x) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\vec x) }{\\partial x_N}, \\frac{\\partial g(\\vec x) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\vec x) }{\\partial x_N^n} \\right) \\right)^2\n", + "c\\left(\\vec{x}, P\\right) = f\\left( \\left( \\vec{x}, \\frac{\\partial g(\\boldsymbol{x) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\boldsymbol{x) }{\\partial x_N}, \\frac{\\partial g(\\boldsymbol{x) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\boldsymbol{x) }{\\partial x_N^n} \\right) \\right)^2\n", "$$" ] }, diff --git a/doc/src/week43/week43.do.txt b/doc/src/week43/week43.do.txt index 1e9a47e10..ddf5fa8d4 100644 --- a/doc/src/week43/week43.do.txt +++ b/doc/src/week43/week43.do.txt @@ -75,12 +75,12 @@ The cost function $c\left(x, P \right)$ can therefore be expressed as c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, , \, g^{(n)}(x)\right)\big)^2 !et -If $N$ inputs are given as a vector $\vec x$ with elements $x_i$ for $i = 1,\dots,N$, +If $N$ inputs are given as a vector $\bm{x}$ with elements $x_i$ for $i = 1,\dots,N$, the cost function becomes !bt \begin{equation} \label{cost} - c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 + c\left(\bm{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2 \end{equation} !et @@ -89,8 +89,8 @@ The neural net should then find some parameters $P$ that minimizes the cost func !split ===== Minimizing the cost function using gradient descent and automatic differentiation ===== -To perform the minimization using gradient descent, the gradient of $c\left(\vec x, P\right)$ is needed. -It might happen so that finding an analytical expression of the gradient of $c(\vec x, P)$ from (ref{cost}) gets too messy, depending on which cost function one desires to use. +To perform the minimization using gradient descent, the gradient of $c\left(\bm{x}, P\right)$ is needed. +It might happen so that finding an analytical expression of the gradient of $c(\bm{x}, P)$ from (ref{cost}) gets too messy, depending on which cost function one desires to use. Luckily, there exists libraries that makes the job for us through automatic differentiation. Automatic differentiation is a method of finding the derivatives numerically with very high precision. @@ -217,25 +217,25 @@ If the neural network evaluates $g_t(x, P)$ at more values for $x$, say $N$ valu \end{equation} !et -Letting $\vec x$ be a vector with elements $x_i$ and $c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2$ denote the cost function, the minimization problem that our network must solve, becomes +Letting $\bm{x}$ be a vector with elements $x_i$ and $c(\bm{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2$ denote the cost function, the minimization problem that our network must solve, becomes !bt -\min_{P} c(\vec x, P) +\min_{P} c(\bm{x}, P) !et In terms of $P_{\text{hidden} }$ and $P_{\text{output} }$, this could also be expressed as $$ -\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\}) +\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\bm{x}, \{P_{\text{hidden} }, P_{\text{output} }\}) $$ !split ===== A possible implementation of a neural network using Autograd ===== -For simplicity, it is assumed that the input is an array $\vec x = (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills (ref{min}). +For simplicity, it is assumed that the input is an array $\bm{x} = (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills (ref{min}). First, the neural network must feed forward the inputs. -This means that $\vec x$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. +This means that $\bm{x}s$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further. The input layer will consist of $N_{\text{input} }$ neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be $N_{\text{hidden} }$. For the $i$-th in the hidden layer with weight $w_i^{\text{hidden} }$ and bias $b_i^{\text{hidden} }$, the weighting from the $j$-th neuron at the input layer is: @@ -330,7 +330,7 @@ The next step is to decide how the parameters should be changed such that they m The chosen cost function for this problem is !bt -c(\vec x, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 +c(\bm{x}, P) = \frac{1}{N} \sum_i \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2 !et In order to minimize the cost function, an optimization method must be chosen. @@ -341,25 +341,25 @@ Here, gradient descent with a constant step size has been chosen. ===== Gradient descent ===== The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum. -In general, the update of some parameters $\vec \omega$ given a cost function defined by some weights $\vec \omega$, $c(\vec x, \vec \omega)$, goes as follows: +In general, the update of some parameters $\bm{\omega}$ given a cost function defined by some weights $\bm{\omega}$, $c(\bm{x}, \bm{\omega})$, goes as follows: !bt -\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega) +\bm{\omega}_{\text{new} } = \bm{\omega} - \lambda \nabla_{\bm{\omega}} c(\bm{x, \bm{\omega}) !et -for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance. +for a number of iterations or until $ \big|\big| \bm{\omega}_{\text{new} } - \bm{\omega} \big|\big|$ becomes smaller than some given tolerance. -The value of $\lambda$ decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$. -The notation $\nabla_{\vec \omega}$ express the gradient with respect to the elements in $\vec \omega$. +The value of $\lambda$ decides how large steps the algorithm must take in the direction of $ \nabla_{\bm{\omega}} c(\bm{x}, \bm{\omega})$. +The notation $\nabla_{\bm{\omega}}$ express the gradient with respect to the elements in $\bm{\omega$. -In our case, we have to minimize the cost function $c(\vec x, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\text{hidden} }$ and for the output layer $P_{\text{output} }$ . +In our case, we have to minimize the cost function $c(\bm{x}, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\text{hidden} }$ and for the output layer $P_{\text{output} }$ . This means that $P_{\text{hidden} }$ and $P_{\text{output} }$ is updated by !bt \begin{aligned} -P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P) \\ -P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P) +P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\bm{x}, P) \\ +P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\bm{x}, P) \end{aligned} !et @@ -2823,10 +2823,10 @@ c\left(x_1, \dots, x_N, P\right) = \left( f\left(x_1, \, \dots \, , x_N, \frac{ \end{equation*} !et -If we let $\vec x = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following: +If we let $\bm{x = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following: !bt \begin{equation*} - c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2 + c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\bm{x) }{\partial x_1}, \dots , \frac{\partial g(\bm{x) }{\partial x_N}, \frac{\partial g(\bm{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\bm{x) }{\partial x_N^n} \right) \right)^2 \end{equation*} !et