Fix minor mistakes and add report for ex 39

report/chapters/methods.tex

@@ -57,7 +57,7 @@ where the $L_1$ norm is defined as $\norm{\vec \theta}_1 = \sum_{i=0}^{p-1} |\th
 \begin{equation} \label{eq:ridge_dof}
     n_\text{effective} = \sum_{i=1}^{n_\text{features}} \frac{d_i^2}{d_i^2 + \lambda},
 \end{equation}
-where $d_i$ are the singular values of the design matrix $X$. Thus regularization effectively reduces the number of parameters in the model, leading to a bias-variance tradeoff that can be tuned by the regularization parameter $\lambda$. While OLS models can be analytically optimized using \cref{eq:ols_solution} for the case where $X^TX$ is invertible, the same is true for Ridge regression in the case where $X^TX + \lambda I$ is invertible. The optimal parameters for Ridge regression can be found by solving the modified normal equations\cite{elstner_lecture_2025}
+where $d_i$ are the singular values of the design matrix $X$. Thus, regularization effectively reduces the number of parameters in the model, leading to a bias-variance tradeoff that can be tuned by the regularization parameter $\lambda$. While OLS models can be analytically optimized using \cref{eq:ols_solution} for the case where $X^TX$ is invertible, the same is true for Ridge regression in the case where $X^TX + \lambda I$ is invertible. The optimal parameters for Ridge regression can be found by solving the modified normal equations\cite{elstner_lecture_2025}
 \begin{equation} \label{eq:ridge_solution}
     \vec \theta_\mathrm{Ridge} = (X^TX + \lambda I)^{-1}X^T \vec y.
 \end{equation}
@@ -179,7 +179,7 @@ The process of k-fold cross-validation is illustrated in \cref{fig:crossvalidati
     (6.8*\squaresize,\ybottom)
     node[midway,xshift=1.5cm]{$\displaystyle \frac{1}{5}\sum_{i=1}^{5}\mathrm{MSE}_i$};
 \end{tikzpicture}
-    \caption{Illustration of 5-fold cross-validation. Each row corresponds to one fold: the turquoise block marks the held-out test set while the white blocks form the training set. The models error $\mathrm{MSE}_i$ is computed for each fold, and the final performance is the average of all $\mathrm{MSE}_i$.}
+    \caption{Illustration of 5-fold cross-validation. Each row corresponds to one fold: the turquoise block marks the held-out test set while the white blocks form the training set. The models' error $\mathrm{MSE}_i$ is computed for each fold, and the final performance is the average of all $\mathrm{MSE}_i$.}
     \label{fig:crossvalidation}
 \end{figure}
 
@@ -189,7 +189,7 @@ The process of k-fold cross-validation is illustrated in \cref{fig:crossvalidati
 
 The methods described in the previous section have been implemented in Python. The implementation is structured in a modular way, allowing to easily switch between different optimization algorithms and resampling methods. The code is available on GitHub at \url{https://github.uio.no/larsbog/FYSSTK-Project1}. The following libraries have been used in the implementation:
 \begin{description}
-    \item[\texttt{numpy}\cite{harris_array_2020}] It is a fundamental package for scientific computing in Python. It provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays efficiently. It is used in this project for all matrix and vector operations, as well as for generating random numbers and performing statistical calculations.
+    \item[\texttt{numpy}\cite{harris_array_2020}] It is a fundamental package for scientific computing in Python. It provides support for large, multidimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays efficiently. It is used in this project for all matrix and vector operations, as well as for generating random numbers and performing statistical calculations.
     \item[\texttt{scikit-learn}\cite{pedregosa_scikit-learn_2011}] It is a machine learning library for Python that provides simple and efficient tools for data mining and data analysis. It includes implementations of various machine learning techniques, like data manipulation and performance evaluation metrics.
     It is used in this project for rescaling the data, splitting the data into training and test sets, and for implementing the resampling methods. Furthermore, it provides the implementation of the mean squared error (MSE) metric used to evaluate the performance of the models.
     \item[\texttt{matplotlib}\cite{hunter_matplotlib_2007}] It is a plotting library for Python that provides a wide range of tools for creating different types of plots and visualizations. It is used in this project to visualize the results of the different methods and to create plots for the report.
@@ -206,6 +206,6 @@ The Runge function is defined as
 \begin{equation} \label{eq:runge_function}
     f(x) = \frac{1}{1 + 25x^2}, \quad x \in [-1, 1].
 \end{equation}
-The $x$-values are sampled uniformly in the interval $[-1, 1]$, and the corresponding $y$-values are generated by adding Gaussian noise with mean 0 and standard deviation $\sigma = 1$, unless otherwise specified. To evaluate the performance in dependance of the degrees of freedom, the $x$-values are expanded into multiple polynomial features up to a specified degree: $X_i = \left\{x_i^0, x_i^1, \dots, x_i^p\right\}$. All features and outputs are then rescaled to have zero mean and unit variance using the \texttt{StandardScaler} from \texttt{scikit-learn}. The mean and variance of the training data are used for rescaling the test data to avoid data leakage. 
-There are analytical implementations of the OLS and Ridge regression methods, as described in \cref{eq:ols_solution} and \cref{eq:ridge_solution}. Furthermore numerical optimizers are implemented from scratch based on a class inheritance structure to reduce code duplication. The base class \texttt{GradientDescent} implements a fitting procedure consisting of a precomputation step, an optimization procedure over multiple iterations and a memory of the cost function values. The different optimization algorithms, i.e. vanilla gradient descent, momentum based gradient descent, Adagrad, RMSProp and Adam, are implemented as subclasses that override the update rule for the parameters $\vec \theta$ in each iteration. The cost functions for OLS, Ridge and Lasso regression are also implemented as separate classes that provide methods to calculate the cost and its gradient. The stochastic gradient descent methods are implemented as further subclasses that modify the data used for fitting in each iteration. In the stochastic gradient descent method, the cost history property is furthermore modified to return the per epoch average of the cost function values. The resampling methods, i.e. bootstrapping and k-fold cross-validation, are implemented as methods with identical interfaces which provide multiple sets of training and test data. This allows to easily switch between the different resampling methods when evaluating the performance of a model.
+The $x$-values are sampled uniformly in the interval $[-1, 1]$, and the corresponding $y$-values are generated by adding Gaussian noise with mean 0 and standard deviation $\sigma = 1$, unless otherwise specified. To evaluate the performance in dependence of the degrees of freedom, the $x$-values are expanded into multiple polynomial features up to a specified degree: $X_i = \left\{x_i^0, x_i^1, \dots, x_i^p\right\}$. All features and outputs are then rescaled to have zero mean and unit variance using the \texttt{StandardScaler} from \texttt{scikit-learn}. The mean and variance of the training data are used for rescaling the test data to avoid data leakage. 
+There are analytical implementations of the OLS and Ridge regression methods, as described in \cref{eq:ols_solution} and \cref{eq:ridge_solution}. Furthermore, numerical optimizers are implemented from scratch based on a class inheritance structure to reduce code duplication. The base class \texttt{GradientDescent} implements a fitting procedure consisting of a precomputation step, an optimization procedure over multiple iterations and a memory of the cost function values. The different optimization algorithms, i.e. vanilla gradient descent, momentum based gradient descent, Adagrad, RMSProp and Adam, are implemented as subclasses that override the update rule for the parameters $\vec \theta$ in each iteration. The cost functions for OLS, Ridge and Lasso regression are also implemented as separate classes that provide methods to calculate the cost and its gradient. The stochastic gradient descent methods are implemented as further subclasses that modify the data used for fitting in each iteration. In the stochastic gradient descent method, the cost history property is furthermore modified to return the per epoch average of the cost function values. The resampling methods, i.e. bootstrapping and k-fold cross-validation, are implemented as methods with identical interfaces which provide multiple sets of training and test data. This allows to easily switch between the different resampling methods when evaluating the performance of a model.