Add some minor sentences

report/chapters/methods.tex

@@ -201,11 +201,19 @@ The LLM ChatGPT-4 has been used to create \cref{fig:crossvalidation}. The LaTeX
 
 
 \subsubsection{Implementation Details}
+\begin{figure}
+    \centering
+    \includegraphics[width=\columnwidth]{../figures/data_scatter.png}
+    \caption{Scatter plot of the synthetic dataset generated using the Runge function with added Gaussian noise. The blue dots represent the test dataset, while the red dots represent the training dataset.}
+    \label{fig:data_scatter}
+\end{figure}
+
 To evaluate the performance of the different methods, a synthetic dataset has been generated using the Runge function.
 The Runge function is defined as
 \begin{equation} \label{eq:runge_function}
     f(x) = \frac{1}{1 + 25x^2}, \quad x \in [-1, 1].
 \end{equation}
-The $x$-values are sampled uniformly in the interval $[-1, 1]$, and the corresponding $y$-values are generated by adding Gaussian noise with mean 0 and standard deviation $\sigma = 1$, unless otherwise specified. To evaluate the performance in dependence of the degrees of freedom, the $x$-values are expanded into multiple polynomial features up to a specified degree: $X_i = \left\{x_i^0, x_i^1, \dots, x_i^p\right\}$. All features and outputs are then rescaled to have zero mean and unit variance using the \texttt{StandardScaler} from \texttt{scikit-learn}. The mean and variance of the training data are used for rescaling the test data to avoid data leakage. 
+The $x$-values are sampled uniformly in the interval $[-1, 1]$, and the corresponding $y$-values are generated by adding Gaussian noise with mean 0 and standard deviation $\sigma = 1$, unless otherwise specified. A visualization of the generated dataset is shown in \cref{fig:data_scatter}. The large amount of noise relative to the domain of $f(x)$ poses a critical challenge to all the models used, as the relation between output and features would otherwise be quite simple for ordinary methods. On the other hand the large amount of noise still allows to derive an optimal parametrization of the model, as the Central Limit Theorem ensures that the mean of the noise converges to zero for large datasets. This allows for a study of big data scenarios, in the limit of simple underlying relations.
+To evaluate the performance in dependence of the degrees of freedom, the $x$-values are expanded into multiple polynomial features up to a specified degree: $X_i = \left\{x_i^0, x_i^1, \dots, x_i^p\right\}$. All features and outputs are then rescaled to have zero mean and unit variance using the \texttt{StandardScaler} from \texttt{scikit-learn}. The mean and variance of the training data are used for rescaling the test data to avoid data leakage. 
 There are analytical implementations of the OLS and Ridge regression methods, as described in \cref{eq:ols_solution} and \cref{eq:ridge_solution}. Furthermore, numerical optimizers are implemented from scratch based on a class inheritance structure to reduce code duplication. The base class \texttt{GradientDescent} implements a fitting procedure consisting of a precomputation step, an optimization procedure over multiple iterations and a memory of the cost function values. The different optimization algorithms, i.e. vanilla gradient descent, momentum based gradient descent, Adagrad, RMSProp and Adam, are implemented as subclasses that override the update rule for the parameters $\vec \theta$ in each iteration. The cost functions for OLS, Ridge and Lasso regression are also implemented as separate classes that provide methods to calculate the cost and its gradient. The stochastic gradient descent methods are implemented as further subclasses that modify the data used for fitting in each iteration. In the stochastic gradient descent method, the cost history property is furthermore modified to return the per epoch average of the cost function values. The resampling methods, i.e. bootstrapping and k-fold cross-validation, are implemented as methods with identical interfaces which provide multiple sets of training and test data. This allows to easily switch between the different resampling methods when evaluating the performance of a model.
 

report/main.tex

@@ -1,4 +1,4 @@
-\documentclass[amsmath,amssymb,twocolumn,aps]{revtex4}
+\documentclass[amsmath,amssymb,twocolumn,aps,rmp]{revtex4}
 
 % allows special characters (including æøå)
 \usepackage[utf8]{inputenc}
@@ -15,6 +15,7 @@
 \usepackage{tikz}
 
 \usetikzlibrary{decorations.pathreplacing}
+\usepackage{csquotes}
 
 \usepackage{xcolor}
 \definecolor{turq}{RGB}{64,224,208}