test

doc/src/week43/week43.do.txt

@@ -75,12 +75,12 @@ The cost function $c\left(x, P \right)$ can therefore be expressed as
 c\left(x, P\right) = \big(f\left(x, \, g(x), \, g'(x), \, g''(x), \, \dots \, , \, g^{(n)}(x)\right)\big)^2
 !et
 
-If $N$ inputs are given as a vector $\vec x$ with elements $x_i$ for $i = 1,\dots,N$,
+If $N$ inputs are given as a vector $\bm{x}$ with elements $x_i$ for $i = 1,\dots,N$,
 the cost function becomes
 
 !bt
 \begin{equation} \label{cost}
-	c\left(\vec x, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
+	c\left(\bm{x}, P\right) = \frac{1}{N} \sum_{i=1}^N \big(f\left(x_i, \, g(x_i), \, g'(x_i), \, g''(x_i), \, \dots \, , \, g^{(n)}(x_i)\right)\big)^2
 \end{equation}
 !et
 
@@ -89,8 +89,8 @@ The neural net should then find some parameters $P$ that minimizes the cost func
 
 !split
 ===== Minimizing the cost function using gradient descent and automatic differentiation =====
-To perform the minimization using gradient descent, the gradient of $c\left(\vec x, P\right)$ is needed.
-It might happen so that finding an analytical expression of the gradient of $c(\vec x, P)$ from (ref{cost}) gets too messy, depending on which cost function one desires to use.
+To perform the minimization using gradient descent, the gradient of $c\left(\bm{x}, P\right)$ is needed.
+It might happen so that finding an analytical expression of the gradient of $c(\bm{x}, P)$ from (ref{cost}) gets too messy, depending on which cost function one desires to use.
 
 Luckily, there exists libraries that makes the job for us through automatic differentiation.
 Automatic differentiation is a method of finding the derivatives numerically with very high precision.
@@ -217,25 +217,25 @@ If the neural network evaluates $g_t(x, P)$ at more values for $x$, say $N$ valu
 \end{equation}
 !et
 
-Letting $\vec x$ be a vector with elements $x_i$ and $c(\vec x, P) = \frac{1}{N} \sum_i  \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2$ denote the cost function, the minimization problem that our network must solve, becomes
+Letting $\bm{x}$ be a vector with elements $x_i$ and $c(\bm{x}, P) = \frac{1}{N} \sum_i  \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2$ denote the cost function, the minimization problem that our network must solve, becomes
 
 !bt
-\min_{P} c(\vec x, P)
+\min_{P} c(\bm{x}, P)
 !et
 
 In terms of $P_{\text{hidden} }$ and $P_{\text{output} }$, this could also be expressed as
 
 $$
-\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\vec x, \{P_{\text{hidden} }, P_{\text{output} }\})
+\min_{P_{\text{hidden} }, \ P_{\text{output} }} c(\bm{x}, \{P_{\text{hidden} }, P_{\text{output} }\})
 $$
 
 !split
 ===== A possible implementation of a neural network using Autograd =====
 
-For simplicity, it is assumed that the input is an array $\vec x = (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills (ref{min}).
+For simplicity, it is assumed that the input is an array $\bm{x} = (x_1, \dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills (ref{min}).
 
 First, the neural network must feed forward the inputs.
-This means that $\vec x$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
+This means that $\bm{x}s$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.
 The input layer will consist of $N_{\text{input} }$ neurons, passing its element to each neuron in the hidden layer.  The number of neurons in the hidden layer will be $N_{\text{hidden} }$.
 
 For the $i$-th in the hidden layer with weight $w_i^{\text{hidden} }$ and bias $b_i^{\text{hidden} }$, the weighting from the $j$-th neuron at the input layer is:
@@ -330,7 +330,7 @@ The next step is to decide how the parameters should be changed such that they m
 The chosen cost function for this problem is
 
 !bt
-c(\vec x, P) = \frac{1}{N} \sum_i  \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
+c(\bm{x}, P) = \frac{1}{N} \sum_i  \big(g_t'(x_i, P) - ( -\gamma g_t(x_i, P) \big)^2
 !et
 
 In order to minimize the cost function, an optimization method must be chosen.
@@ -341,25 +341,25 @@ Here, gradient descent with a constant step size has been chosen.
 ===== Gradient descent =====
 The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
 
-In general, the update of some parameters $\vec \omega$ given a cost function defined by some weights $\vec \omega$, $c(\vec x, \vec \omega)$, goes as follows:
+In general, the update of some parameters $\bm{\omega}$ given a cost function defined by some weights $\bm{\omega}$, $c(\bm{x}, \bm{\omega})$, goes as follows:
 
 !bt
-\vec \omega_{\text{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(\vec x, \vec \omega)
+\bm{\omega}_{\text{new} } = \bm{\omega} - \lambda \nabla_{\bm{\omega}} c(\bm{x, \bm{\omega})
 !et
 
-for a number of iterations or until $ \big|\big| \vec \omega_{\text{new} } - \vec \omega \big|\big|$ becomes smaller than some given tolerance.
+for a number of iterations or until $ \big|\big| \bm{\omega}_{\text{new} } - \bm{\omega} \big|\big|$ becomes smaller than some given tolerance.
 
-The value of $\lambda$ decides how large steps the algorithm must take in the direction of $ \nabla_{\vec \omega} c(\vec x, \vec \omega)$.
-The notation $\nabla_{\vec \omega}$ express the gradient with respect to the elements in $\vec \omega$.
+The value of $\lambda$ decides how large steps the algorithm must take in the direction of $ \nabla_{\bm{\omega}} c(\bm{x}, \bm{\omega})$.
+The notation $\nabla_{\bm{\omega}}$ express the gradient with respect to the elements in $\bm{\omega$.
 
-In our case, we have to minimize the cost function $c(\vec x, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\text{hidden} }$ and for the output layer $P_{\text{output} }$ .
+In our case, we have to minimize the cost function $c(\bm{x}, P)$ with respect to the two sets of weights and biases, that is for the hidden layer $P_{\text{hidden} }$ and for the output layer $P_{\text{output} }$ .
 
 This means that $P_{\text{hidden} }$ and $P_{\text{output} }$ is updated by
 
 !bt
 \begin{aligned}
-P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\vec x, P)  \\
-P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\vec x, P)
+P_{\text{hidden},\text{new}} &= P_{\text{hidden}} - \lambda \nabla_{P_{\text{hidden}}} c(\bm{x}, P)  \\
+P_{\text{output},\text{new}} &= P_{\text{output}} - \lambda \nabla_{P_{\text{output}}} c(\bm{x}, P)
 \end{aligned}
 !et
 
@@ -2823,10 +2823,10 @@ c\left(x_1, \dots, x_N, P\right) = \left(  f\left(x_1, \, \dots \, , x_N, \frac{
 \end{equation*}
 !et
 
-If we let $\vec x = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following:
+If we let $\bm{x = \big( x_1, \dots, x_N \big)$ be an array containing the values for $x_1, \dots, x_N$ respectively, the cost function can be reformulated into the following:
 !bt
 \begin{equation*}
-	c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\vec x) }{\partial x_1}, \dots , \frac{\partial g(\vec x) }{\partial x_N}, \frac{\partial g(\vec x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\vec x) }{\partial x_N^n} \right) \right)^2
+	c\left(\vec{x}, P\right) = f\left( \left( \vec{x}, \frac{\partial g(\bm{x) }{\partial x_1}, \dots , \frac{\partial g(\bm{x) }{\partial x_N}, \frac{\partial g(\bm{x) }{\partial x_1\partial x_2}, \, \dots \, , \frac{\partial^n g(\bm{x) }{\partial x_N^n} \right) \right)^2
 \end{equation*}
 !et