lars/FYSSTK-Project1
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Hopefully I'm done

Browse Source
This commit is contained in:
lars
2025-10-02 19:01:46 +02:00
parent f000747a0d
commit 2e6723813d
6 changed files with 33 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
report/chapters/methods.tex
+11 -11
View File
@@ -15,7 +15,7 @@ To find the optimal solution to a linear model in the form
\begin{equation} \label{eq:linear_model}
X \vec \theta = \vec y, \quad X \in \mathbb{R}^{n\times p}, \vec y \in \mathbb{R}^n, \vec \theta \in \mathbb{R}^p,
\end{equation}
the ordinary least squares (OLS) method can be used. The OLS method minimizes the squared deviation between the model prediction and the actual data\cite{hastie_elements_2009}. This is done by minimizing the cost function
the ordinary least squares (OLS) method can be used. The OLS method minimizes the squared deviation between the model prediction and the actual data \cite{hastie_elements_2009}. This is done by minimizing the cost function
\begin{equation} \label{eq:ols_cost}
\cols(\vec \theta) = \frac{1}{n} \norm{X\vec \theta - \vec y}_2^2.
\end{equation}
@@ -45,7 +45,7 @@ and the variance as
\begin{equation} \label{eq:variance}
\var(\vec{\tilde y}) = \E\left[(\vec{\tilde y} - \E[\vec{\tilde y}])^2\right].
\end{equation}
The irreducible error $\sigma^2$ is the variance of the noise in the data and cannot be reduced by any model. The bias-variance decomposition in \cref{eq:bias_variance_decomp} shows that there is a tradeoff between bias and variance when trying to minimize the MSE. A model with low complexity will have a high bias but low variance, while a model with high complexity will have a low bias but high variance. This is known as the bias-variance tradeoff. The OLS method can lead to overfitting of the data, especially when the number of parameters $p$ is large compared to the number of data points $n$. In this case, the model will fit the noise in the data rather than the underlying trend, leading to a low bias but high variance. On the contrary, a model with too few parameters will not be able to capture the underlying trend in the data, leading to a high bias but low variance\cite{elstner_lecture_2025,hastie_elements_2009}. To mitigate the effects of overfitting, regularization techniques such as Ridge and Lasso regression can be used. These techniques add a penalty term to the cost function that penalizes large values of the parameters $\vec \theta$\cite{lekhansh_lasso_2024}. The cost function for Ridge regression is given by
The irreducible error $\sigma^2$ is the variance of the noise in the data and cannot be reduced by any model. The bias-variance decomposition in \cref{eq:bias_variance_decomp} shows that there is a tradeoff between bias and variance when trying to minimize the MSE. A model with low complexity will have a high bias but low variance, while a model with high complexity will have a low bias but high variance. This is known as the bias-variance tradeoff. The OLS method can lead to overfitting of the data, especially when the number of parameters $p$ is large compared to the number of data points $n$. In this case, the model will fit the noise in the data rather than the underlying trend, leading to a low bias but high variance. On the contrary, a model with too few parameters will not be able to capture the underlying trend in the data, leading to a high bias but low variance \cite{elstner_lecture_2025,hastie_elements_2009}. To mitigate the effects of overfitting, regularization techniques such as Ridge and Lasso regression can be used. These techniques add a penalty term to the cost function that penalizes large values of the parameters $\vec \theta$\cite{lekhansh_lasso_2024}. The cost function for Ridge regression is given by
\begin{equation} \label{eq:ridge_cost}
\colr(\vec \theta) = \frac{1}{n} \norm{X\vec \theta - \vec y}_2^2 + \lambda \norm{\vec \theta}_2^2,
\end{equation}
@@ -53,11 +53,11 @@ where $\lambda$ is the regularization parameter that controls the strength of th
\begin{equation} \label{eq:lasso_cost}
\coll(\vec \theta) = \frac{1}{n} \norm{X\vec \theta - \vec y}_2^2 + \lambda \norm{\vec \theta}_1,
\end{equation}
where the $L_1$ norm is defined as $\norm{\vec \theta}_1 = \sum_{i=0}^{p-1} |\theta_i|$\cite{lekhansh_lasso_2024,elstner_lecture_2025}. Both Ridge and Lasso regression can help to reduce the variance of the model by penalizing large values of the parameters $\vec \theta$, leading to a different bias-variance tradeoff. For Ridge regression, we can derive the effective number of parameters also called degrees of freedom as \cite{hastie_elements_2009}
where the $L_1$ norm is defined as $\norm{\vec \theta}_1 = \sum_{i=0}^{p-1} |\theta_i|$ \cite{lekhansh_lasso_2024,elstner_lecture_2025}. Both Ridge and Lasso regression can help to reduce the variance of the model by penalizing large values of the parameters $\vec \theta$, leading to a different bias-variance tradeoff. For Ridge regression, we can derive the effective number of parameters also called degrees of freedom as \cite{hastie_elements_2009}
\begin{equation} \label{eq:ridge_dof}
n_\text{effective} = \sum_{i=1}^{n_\text{features}} \frac{d_i^2}{d_i^2 + \lambda},
\end{equation}
where $d_i$ are the singular values of the design matrix $X$. Thus, regularization effectively reduces the number of parameters in the model, leading to a bias-variance tradeoff that can be tuned by the regularization parameter $\lambda$. While OLS models can be analytically optimized using \cref{eq:ols_solution} for the case where $X^TX$ is invertible, the same is true for Ridge regression in the case where $X^TX + \lambda I$ is invertible. The optimal parameters for Ridge regression can be found by solving the modified normal equations\cite{elstner_lecture_2025}
where $d_i$ are the singular values of the design matrix $X$. Thus, regularization effectively reduces the number of parameters in the model, leading to a bias-variance tradeoff that can be tuned by the regularization parameter $\lambda$. While OLS models can be analytically optimized using \cref{eq:ols_solution} for the case where $X^TX$ is invertible, the same is true for Ridge regression in the case where $X^TX + \lambda I$ is invertible. The optimal parameters for Ridge regression can be found by solving the modified normal equations \cite{elstner_lecture_2025}
\begin{equation} \label{eq:ridge_solution}
\vec \theta_\mathrm{Ridge} = (X^TX + \lambda I)^{-1}X^T \vec y.
\end{equation}
@@ -118,7 +118,7 @@ The Adam (Adaptive Moment Estimation) algorithm combines the ideas of momentum a
where $m_i$ is the moving average of the gradients, $s_i$ is the moving average of the squared gradients, $\beta_1$ and $\beta_2$ are hyperparameters that control the decay rates of the moving averages, and the other symbols have the same meaning as in the previous update rules \cite{goodfellow_deep_2016,noauthor_stochastic_2025}.
\subsubsection{Stochastic Gradient Descent}
As the computation of the gradient $\nabla \cols(\vec \theta_i)$ can be computationally expensive, especially for large datasets, a common approach is to use stochastic gradient descent (SGD). In SGD, the gradient is approximated by calculating it on a small subset of the data, called a mini-batch. This leads to a noisy estimate of the gradient, but it can significantly reduce the computational cost of each update step. The main challenge with SGD is to choose an appropriate mini-batch size. A small mini-batch size leads to a noisy estimate of the gradient, which can slow down convergence. On the other hand, a large mini-batch size can diminish the benefits of using SGD, as it approaches the full gradient descent computation cost. In practice the optimal mini-batch size depends on the specific problem and dataset\cite{goodfellow_deep_2016,elstner_lecture_2025}.
As the computation of the gradient $\nabla \cols(\vec \theta_i)$ can be computationally expensive, especially for large datasets, a common approach is to use stochastic gradient descent (SGD). In SGD, the gradient is approximated by calculating it on a small subset of the data, called a mini-batch. This leads to a noisy estimate of the gradient, but it can significantly reduce the computational cost of each update step. The main challenge with SGD is to choose an appropriate mini-batch size. A small mini-batch size leads to a noisy estimate of the gradient, which can slow down convergence. On the other hand, a large mini-batch size can diminish the benefits of using SGD, as it approaches the full gradient descent computation cost. In practice the optimal mini-batch size depends on the specific problem and dataset \cite{goodfellow_deep_2016,elstner_lecture_2025}.
While the high variance of models with a high number of parameters can be counteracted with a large amount of data, the computational cost of fitting such models can become very high. This is especially relevant in the context of big data, where datasets can contain millions or even billions of samples. In such cases, it can be infeasible to use optimization algorithms that require the computation of the gradient over the entire dataset, such as ordinary gradient descent. For a dataset with $\N$ samples and a model with $p$ parameters, the computational cost of calculating the gradient
\begin{equation}
@@ -127,14 +127,14 @@ While the high variance of models with a high number of parameters can be counte
is of order $\mathcal{O}(N p^2)$, as the matrix-vector products $X^T X$ needs to be computed. For very large datasets this can lead to very long computation times, as the entire dataset needs to be processed to compute the gradient. A common approach to counteract this problem is to use stochastic gradient descent (SGD) or mini-batch gradient descent, where the gradient is computed over a small subset of the data, called a batch. This leads to a significant reduction in computational cost, as the gradient can be computed in $\mathcal{O}(B p^2)$, where $B$ is the batch size. However, this comes at the cost of increased variance in the gradient estimate, as the gradient is only computed over a small subset of the data. This can lead to slower convergence and more oscillations in the optimization process.
While the decrease in computation time is significant, the result is even less black and white, when taking a closer look at the details.
While conventional GD can benefit largely by precomputing the matrix products $X^T X$ and $X^T \vec{y}$, this is not possible for SGD, as the batches change in every epoch. This leads to a significant increase in computation time per epoch for SGD compared to GD. However, as the number of epochs needed to reach convergence is usually much lower for SGD than for GD, the overall computation time can still be significantly lower for SGD. This is especially relevant for very large datasets, where the computational cost of GD can become prohibitive. Because while GD needs to process the entire dataset and thus the computational cost is of order $\mathcal{O}(N p^2)$, SGD only needs to process a small batch of size $B$ and thus the computational cost is of order $\mathcal{O}(B p^2)$ and invariant of the size of the dataset, given the batch size is large enough to estimate the complete gradient well enough. This leads to a significant reduction in computation time for very large datasets, as the computational cost of SGD does not increase with the size of the dataset\cite{goodfellow_deep_2016}.
While conventional GD can benefit largely by precomputing the matrix products $X^T X$ and $X^T \vec{y}$, this is not possible for SGD, as the batches change in every epoch. This leads to a significant increase in computation time per epoch for SGD compared to GD. However, as the number of epochs needed to reach convergence is usually much lower for SGD than for GD, the overall computation time can still be significantly lower for SGD. This is especially relevant for very large datasets, where the computational cost of GD can become prohibitive. Because while GD needs to process the entire dataset and thus the computational cost is of order $\mathcal{O}(N p^2)$, SGD only needs to process a small batch of size $B$ and thus the computational cost is of order $\mathcal{O}(B p^2)$ and invariant of the size of the dataset, given the batch size is large enough to estimate the complete gradient well enough. This leads to a significant reduction in computation time for very large datasets, as the computational cost of SGD does not increase with the size of the dataset \cite{goodfellow_deep_2016}.
\subsubsection{Resampling Methods}
To better understand the performance of a model, e.g. in terms of the bias-variance tradeoff, resampling methods such as bootstrapping and k-fold cross-validation can be used. These methods allow to estimate the performance of a model on unseen data by repeatedly splitting the data into training and test sets.
\paragraph{Bootstrapping}
In bootstrapping, multiple datasets are generated by randomly sampling the original training dataset with replacement. Each of these datasets is then used to train a model, and the performance of the model is evaluated on the unmodified test dataset. This process is repeated multiple times, and the performance metrics are averaged to obtain an estimate of the model's performance. Bootstrapping can be used to estimate the bias and variance of a model by analyzing the distribution of the performance metrics across the different bootstrap samples\cite{hastie_elements_2009}. The number of bootstrap samples is arbitrary, but a thousand samples are used in this work.
In bootstrapping, multiple datasets are generated by randomly sampling the original training dataset with replacement. Each of these datasets is then used to train a model, and the performance of the model is evaluated on the unmodified test dataset. This process is repeated multiple times, and the performance metrics are averaged to obtain an estimate of the model's performance. Bootstrapping can be used to estimate the bias and variance of a model by analyzing the distribution of the performance metrics across the different bootstrap samples \cite{hastie_elements_2009}. The number of bootstrap samples is arbitrary, but a thousand samples are used in this work.
\paragraph{k-Fold Cross-Validation}
In k-fold cross-validation, the original dataset is divided into $k$ equally sized folds. The model is then trained on $k-1$ folds and evaluated on the remaining fold. This process is repeated $k$ times, with each fold being used as the test set once. The performance metrics are then averaged across the $k$ iterations to obtain an estimate of the model's performance\cite{elstner_lecture_2025}.
In k-fold cross-validation, the original dataset is divided into $k$ equally sized folds. The model is then trained on $k-1$ folds and evaluated on the remaining fold. This process is repeated $k$ times, with each fold being used as the test set once. The performance metrics are then averaged across the $k$ iterations to obtain an estimate of the model's performance \cite{elstner_lecture_2025}.
The process of k-fold cross-validation is illustrated in \cref{fig:crossvalidation}. For this work $k=5$ folds are used.
\begin{figure}
@@ -189,10 +189,10 @@ The process of k-fold cross-validation is illustrated in \cref{fig:crossvalidati
The methods described in the previous section have been implemented in Python. The implementation is structured in a modular way, allowing to easily switch between different optimization algorithms and resampling methods. The code is available on GitHub at \url{https://github.uio.no/larsbog/FYSSTK-Project1}. The following libraries have been used in the implementation:
\begin{description}
\item[\texttt{numpy}\cite{harris_array_2020}] It is a fundamental package for scientific computing in Python. It provides support for large, multidimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays efficiently. It is used in this project for all matrix and vector operations, as well as for generating random numbers and performing statistical calculations.
\item[\texttt{scikit-learn}\cite{pedregosa_scikit-learn_2011}] It is a machine learning library for Python that provides simple and efficient tools for data mining and data analysis. It includes implementations of various machine learning techniques, like data manipulation and performance evaluation metrics.
\item[\texttt{numpy} \cite{harris_array_2020}] It is a fundamental package for scientific computing in Python. It provides support for large, multidimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays efficiently. It is used in this project for all matrix and vector operations, as well as for generating random numbers and performing statistical calculations.
\item[\texttt{scikit-learn} \cite{pedregosa_scikit-learn_2011}] It is a machine learning library for Python that provides simple and efficient tools for data mining and data analysis. It includes implementations of various machine learning techniques, like data manipulation and performance evaluation metrics.
It is used in this project for rescaling the data, splitting the data into training and test sets, and for implementing the resampling methods. Furthermore, it provides the implementation of the mean squared error (MSE) metric used to evaluate the performance of the models.
\item[\texttt{matplotlib}\cite{hunter_matplotlib_2007}] It is a plotting library for Python that provides a wide range of tools for creating different types of plots and visualizations. It is used in this project to visualize the results of the different methods and to create plots for the report.
\item[\texttt{matplotlib} \cite{hunter_matplotlib_2007}] It is a plotting library for Python that provides a wide range of tools for creating different types of plots and visualizations. It is used in this project to visualize the results of the different methods and to create plots for the report.
\end{description}
\subsubsection{Other Software Tools}