lars/FYSSTK-Project1
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Write a significant portion of the report

Browse Source
This commit is contained in:
Lars Bogner
2025-09-17 10:25:09 +02:00
parent 1eaa75a9cf
commit e20075fc4e
6 changed files with 256 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
report/chapters/introduction.tex
+8 -1
View File
@@ -1 +1,8 @@
Test, Test Test
Optimizing numerical models to model a set of data has been a challenge of the natural sciences for many decades. But with the introduction of artificial neural networks in all disciplines over the last years a special focus on those optimization techniques is relevant.
In this paper different methods of such optimization are studied in the context of large datasets and big data with possibly noise-affected input data. To render the best possible fits a multitude of cost functions, minimization algorithms and other approaches to reduce the computation associated with the optimization process are studied and evaluated.
In the following section the theoretical background of the methods used in this paper is presented. In \cref{sec:methods} secondly the implementation of these methods is discussed. The results of applying these methods to different datasets are presented in \cref{sec:results}. Finally, a conclusion is drawn in \cref{sec:conclusion} and an outlook on possible future work is given in \cref{sec:perspective}.
In a first step, the dependence of the bias-variance tradeoff on the number of degrees of freedom in a model is studied. For this purpose, polynomial models of different degrees are fitted to data generated from the Runge function with significant noise. The results are then compared to different Ridge and Lasso models to study the effect of regularization on the bias-variance tradeoff. In the context of big data, the performance of different numerical minimization algorithms is studied. For this purpose, the performance of gradient descent, stochastic gradient descent and different optimization algorithms such as Adam are compared in the context of Ridge and Lasso regression. Furthermore the possible advantages and disadvantages of stochastic gradient descent techniques are evaluated. Finally, the effect of resampling methods such as bootstrapping and k-fold cross-validation on the bias-variance tradeoff is studied.
report/chapters/methods.tex
+152
View File
@@ -0,0 +1,152 @@
\subsection{Theoretical Background}
% - OLS for linear model, analytical solution
% - Bias variance tradeoff derivation from OLS
% - How number of dof effects the bias variance tradeoff
% - Bias variance tradeoff leads to penalty terms for large parameters in the model: Ridge, Lasso
% - Effective number of parameters of ridge (see ex.w.36 )
% - Minimization of analytically unsolvable cost problems requiring numerical approach
% - Minimization via Taylor expansion of multidim function and hessian -> Gradient Descent
% - maybe deriv. of 3 cost functions
% - Other numerical minimization algos: Momentum, Adagrad, RMSProp, Adam
% - Problem of expensive matrix multiplications (order of flops), gradient of subsample approaches gradient of entire sample in the limit -> stochastic gradient descent
% - Intro to bootstrapping and k-fold crossvalidation
To find the optimal solution to a linear model in the form
\begin{equation} \label{eq:linear_model}
X \vec \theta = \vec y, \quad X \in \mathbb{R}^{n\times p}, \vec y \in \mathbb{R}^n, \vec \theta \in \mathbb{R}^p,
\end{equation}
the ordinary least squares (OLS) method can be used. The OLS method minimizes the squared deviation between the model prediction and the actual data. This is done by minimizing the cost function
\begin{equation} \label{eq:ols_cost}
\cols(\vec \theta) = \frac{1}{n} \norm{X\vec \theta - \vec y}_2^2.
\end{equation}
This method has been widely used in the past ultimately also because it has an analytical solution in the case that the matrix product $X^TX$ is invertible, i.e. $\det(X^TX) \neq 0$. In this case the optimal parameters $\vec \theta_{OLS}$ can be found by solving the normal equations
\begin{equation} \label{eq:ols_solution}
\vec \theta_{OLS} = (X^TX)^{-1}X^T \vec y.
\end{equation}
Taking the derivative of the cost function in \cref{eq:ols_cost} with respect to the parameters $\vec \theta$ and inserting the solution in \cref{eq:ols_solution} shows that this is indeed a minimum of the cost function: \todo{Expand on the derivation here?}
\begin{equation}
\frac{\partial \cols}{\partial \vec \theta}\Big\rvert_{\vec \theta_{OLS}} = \frac{2}{n} X^T(X\vec \theta - \vec y)\Big\rvert_{\vec \theta_{OLS}} = 0.
\end{equation}
\subsubsection{Bias-Variance Tradeoff and Regularization}
The cost function of the ordinary least squares method can also be identified as the mean squared error (MSE) between the model prediction and the actual data
\begin{equation} \label{eq:mse}
\mse(\vec y, \vec{\tilde y}) = \frac{1}{n} \sum_{i=0}^{n-1} (y_i - \tilde y_i)^2 = \E \left[(y - \tilde y)^2\right],
\end{equation}
where $\vec{\tilde y} = X \vec \theta$ is the model prediction. $\E$ denotes the expectation value. The MSE can be decomposed into three components, the squared bias, the variance and the irreducible error $\sigma^2$ as
\begin{equation} \label{eq:bias_variance_decomp}
\mse(\vec y, \vec{\tilde y}) = \bias^2(\vec y, \vec{\tilde y}) + \var(\vec{\tilde y}) + \sigma^2,
\end{equation}
where the bias is defined as
\begin{equation} \label{eq:bias}
\bias(\vec y, \vec{\tilde y}) = \E[\vec y] - \E[\vec{\tilde y}],
\end{equation}
and the variance as
\begin{equation} \label{eq:variance}
\var(\vec{\tilde y}) = \E\left[(\vec{\tilde y} - \E[\vec{\tilde y}])^2\right].
\end{equation}
The irreducible error $\sigma^2$ is the variance of the noise in the data and cannot be reduced by any model. The bias-variance decomposition in \cref{eq:bias_variance_decomp} shows that there is a tradeoff between bias and variance when trying to minimize the MSE. A model with low complexity will have a high bias but low variance, while a model with high complexity will have a low bias but high variance. This is known as the bias-variance tradeoff. The OLS method can lead to overfitting of the data, especially when the number of parameters $p$ is large compared to the number of data points $n$. In this case, the model will fit the noise in the data rather than the underlying trend, leading to a low bias but high variance. On the contrary, a model with too few parameters will not be able to capture the underlying trend in the data, leading to a high bias but low variance. To mitigate the effects of overfitting, regularization techniques such as Ridge and Lasso regression can be used. These techniques add a penalty term to the cost function that penalizes large values of the parameters $\vec \theta$. The cost function for Ridge regression is given by
\begin{equation} \label{eq:ridge_cost}
\colr(\vec \theta) = \frac{1}{n} \norm{X\vec \theta - \vec y}_2^2 + \lambda \norm{\vec \theta}_2^2,
\end{equation}
where $\lambda$ is the regularization parameter that controls the strength of the penalty term. The cost function for Lasso regression is given by
\begin{equation} \label{eq:lasso_cost}
\coll(\vec \theta) = \frac{1}{n} \norm{X\vec \theta - \vec y}_2^2 + \lambda \norm{\vec \theta}_1,
\end{equation}
where the $L_1$ norm is defined as $\norm{\vec \theta}_1 = \sum_{i=0}^{p-1} |\theta_i|$. Both Ridge and Lasso regression can help to reduce the variance of the model by penalizing large values of the parameters $\vec \theta$, leading to a different bias-variance tradeoff. For Ridge regression, we can derive the effective number of parameters also called degrees of freedom as
\begin{equation} \label{eq:ridge_dof}
n_\text{effective} = \sum_{i=1}^{n_\text{features}} \frac{d_i^2}{d_i^2 + \lambda},
\end{equation}
where $d_i$ are the singular values of the design matrix $X$. Thus regularization effectively reduces the number of parameters in the model, leading to a bias-variance tradeoff that can be tuned by the regularization parameter $\lambda$. While OLS models can be analytically optimized using \cref{eq:ols_solution} for the case where $X^TX$ is invertible, the same is true for Ridge regression in the case where $X^TX + \lambda I$ is invertible. The optimal parameters for Ridge regression can be found by solving the modified normal equations
\begin{equation} \label{eq:ridge_solution}
\vec \theta_\mathrm{Ridge} = (X^TX + \lambda I)^{-1}X^T \vec y.
\end{equation}
\subsubsection{Numerical Optimization}
However, for Lasso regression there is no analytical solution and numerical methods must be used to obtain the optimal parameters $\vec \theta_\mathrm{Lasso}$. All numerical optimization methods are based on the idea of iteratively minimizing the cost function. Around the current guess for the parameters $\vec \theta_i$ the cost function can be approximated by a Taylor expansion as
\begin{equation} \label{eq:taylor_expansion}
\begin{aligned}
\cols(\vec \theta) &= \cols(\vec \theta_i) + \nabla \cols(\vec \theta_i)^T (\vec \theta - \vec \theta_i) \\&\quad+ \frac{1}{2} (\vec \theta - \vec \theta_i)^T H(\vec \theta_i) (\vec \theta - \vec \theta_i) \\&\quad+ \mathcal{O}\left(\norm{\vec \theta - \vec \theta_i}_2^3\right),
\end{aligned}
\end{equation}
where $\nabla \cols(\vec \theta_i)$ is the gradient and $H(\vec \theta_i)$ the Hessian of the cost function at the current guess $\vec \theta_i$. Based on this approximation, the next guess for the parameters can be found by minimizing the quadratic approximation of the cost function. Since the calculation of the Hessian is computationally expensive, it is often neglected in practice. This leads to the gradient descent algorithm, where the next guess for the parameters is given by
\begin{equation} \label{eq:gradient_descent}
\vec \theta_{i+1} = \vec \theta_i - \eta \nabla \cols(\vec \theta_i),
\end{equation}
where $\eta$ is the learning rate that controls the step size of the update. The gradient descent algorithm can be further improved by using techniques such as momentum, Adagrad, RMSProp and Adam. These techniques adapt the learning rate based on the history of the gradients, leading to faster convergence and better performance.
\paragraph{Adagrad}
The Adagrad algorithm adapts the learning rate for each parameter based on the history of the gradients. The update rule for Adagrad is given by
\begin{equation} \label{eq:adagrad}
\begin{aligned}
g_{i} &= g_{i-1} + \nabla \cols(\vec \theta_i) \odot \nabla \cols(\vec \theta_i), \\
\vec \theta_{i+1} &= \vec \theta_i - \frac{\eta}{\sqrt{g_i + \epsilon}} \odot \nabla \cols(\vec \theta_i),
\end{aligned}
\end{equation}
where $g_i$ is the accumulated squared gradient, $\odot$ denotes the element-wise product and $\epsilon$ is a small constant to prevent division by zero.
\paragraph{RMSProp}
The RMSProp algorithm is a modification of Adagrad that aims to reduce its aggressive, monotonically decreasing learning rate. It does this by maintaining a moving average of the squared gradients, which allows it to adapt the learning rate for each parameter more effectively. The update rule for RMSProp is given by
\begin{equation} \label{eq:rmsprop}
\begin{aligned}
s_{i} &= \beta s_{i-1} + (1 - \beta) \nabla \cols(\vec \theta_i) \odot \nabla \cols(\vec \theta_i), \\
\vec \theta_{i+1} &= \vec \theta_i - \frac{\eta}{\sqrt{s_i + \epsilon}} \odot \nabla \cols(\vec \theta_i),
\end{aligned}
\end{equation}
where $s_i$ is the moving average of the squared gradients, $\beta$ is a hyperparameter that controls the decay rate of the moving average, and the other symbols have the same meaning as in the Adagrad update rule.
\paragraph{Adam}
The Adam (Adaptive Moment Estimation) algorithm combines the ideas of momentum and RMSProp. It maintains a moving average of both the gradients and the squared gradients, allowing it to adapt the learning rate for each parameter based on both the first and second moments of the gradients. The update rule for Adam is given by
\begin{equation} \label{eq:adam}
\begin{aligned}
m_{i} &= \beta_1 m_{i-1} + (1 - \beta_1) \nabla \cols(\vec \theta_i), \\
s_{i} &= \beta_2 s_{i-1} + (1 - \beta_2) \nabla \cols(\vec \theta_i) \odot \nabla \cols(\vec \theta_i), \\
\vec \theta_{i+1} &= \vec \theta_i - \frac{\eta}{\sqrt{s_i + \epsilon}} \odot m_i,
\end{aligned}
\end{equation}
where $m_i$ is the moving average of the gradients, $s_i$ is the moving average of the squared gradients, $\beta_1$ and $\beta_2$ are hyperparameters that control the decay rates of the moving averages, and the other symbols have the same meaning as in the previous update rules.
\subsubsection{Stochastic Gradient Descent}
As the computation of the gradient $\nabla \cols(\vec \theta_i)$ can be computationally expensive, especially for large datasets, a common approach is to use stochastic gradient descent (SGD). In SGD, the gradient is approximated by calculating it on a small subset of the data, called a mini-batch. This leads to a noisy estimate of the gradient, but it can significantly reduce the computational cost of each update step. The main challenge with SGD is to choose an appropriate mini-batch size. A small mini-batch size leads to a noisy estimate of the gradient, which can slow down convergence. On the other hand, a large mini-batch size can diminish the benefits of using SGD, as it approaches the full gradient descent computation cost. In practice the optimal mini-batch size depends on the specific problem and dataset.
\subsubsection{Resampling Methods}
To better understand the performance of a model, e.g. in terms of the bias-variance tradeoff, resampling methods such as bootstrapping and k-fold cross-validation can be used. These methods allow to estimate the performance of a model on unseen data by repeatedly splitting the data into training and test sets.
\paragraph{Bootstrapping}
In bootstrapping, multiple datasets are generated by randomly sampling the original training dataset with replacement. Each of these datasets is then used to train a model, and the performance of the model is evaluated on the unmodified test dataset. This process is repeated multiple times, and the performance metrics are averaged to obtain an estimate of the model's performance. Bootstrapping can be used to estimate the bias and variance of a model by analyzing the distribution of the performance metrics across the different bootstrap samples. The number of bootstrap samples is arbitrary, but a thousand samples are used in this work.
\paragraph{k-Fold Cross-Validation}
In k-fold cross-validation, the original dataset is divided into $k$ equally sized folds. The model is then trained on $k-1$ folds and evaluated on the remaining fold. This process is repeated $k$ times, with each fold being used as the test set once. The performance metrics are then averaged across the $k$ iterations to obtain an estimate of the model's performance.
The process of k-fold cross-validation is illustrated in \cref{fig:crossvalidation}. For this work $k=5$ folds are used.
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{../figures/crossvalidation.png}
\caption{Illustration of k-fold cross-validation with $k=5$. The dataset is divided into 5 folds, and the model is trained on 4 folds and evaluated on the remaining fold. This process is repeated 5 times, with each fold being used as the test set once.}
\label{fig:crossvalidation}
\end{figure}
\subsection{Software Tools and Implementation}
\subsubsection{Programming Language and Libraries}
The methods described in the previous section have been implemented in Python. The implementation is structured in a modular way, allowing to easily switch between different optimization algorithms and resampling methods. The code is available on GitHub at \url{https://github.uio.no/larsbog/FYSSTK-Project1}. The following libraries have been used in the implementation:
\begin{description}
\item[\texttt{numpy}] It is a fundamental package for scientific computing in Python. It provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays efficiently. It is used in this project for all matrix and vector operations, as well as for generating random numbers and performing statistical calculations.
\item[\texttt{scikit-learn}] It is a machine learning library for Python that provides simple and efficient tools for data mining and data analysis. It includes implementations of various machine learning techniques, like data manipulation and performance evaluation metrics.
It is used in this project for rescaling the data, splitting the data into training and test sets, and for implementing the resampling methods. Furthermore, it provides the implementation of the mean squared error (MSE) metric used to evaluate the performance of the models.
\item[\texttt{matplotlib}] It is a plotting library for Python that provides a wide range of tools for creating different types of plots and visualizations. It is used in this project to visualize the results of the different methods and to create plots for the report.
\end{description}
\subsubsection{Other Software Tools}
The code has been developed using the \texttt{Visual Studio Code} editor, which provides a wide range of features for Python development, including syntax highlighting, code completion, and debugging tools. The version control system \texttt{Git} has been used to manage the codebase and track changes. The Large Language Model (LLM) \texttt{GitHub Copilot} has been used to assist in the development of the code by providing code suggestions and autocompletions based on the context of the code being written. All LLM-generated code has been reviewed and modified as necessary to ensure its correctness and suitability for the project.
\subsubsection{Implementation Details}
To evaluate the performance of the different methods, a synthetic dataset has been generated using the Runge function.
The Runge function is defined as
\begin{equation} \label{eq:runge_function}
f(x) = \frac{1}{1 + 25x^2}, \quad x \in [-1, 1].
\end{equation}
The $x$-values are sampled uniformly in the interval $[-1, 1]$, and the corresponding $y$-values are generated by adding Gaussian noise with mean 0 and standard deviation $\sigma = 1$, unless otherwise specified. To evaluate the performance in dependance of the degrees of freedom, the $x$-values are expanded into multiple polynomial features up to a specified degree: $X_i = \left\{x_i^0, x_i^1, \dots, x_i^p\right\}$. All features and outputs are then rescaled to have zero mean and unit variance using the \texttt{StandardScaler} from \texttt{scikit-learn}. The mean and variance of the training data are used for rescaling the test data to avoid data leakage.
There are analytical implementations of the OLS and Ridge regression methods, as described in \cref{eq:ols_solution} and \cref{eq:ridge_solution}. Furthermore numerical optimizers are implemented from scratch based on a class inheritance structure to reduce code duplication. The base class \texttt{GradientDescent} implements a fitting procedure consisting of a precomputation step, an optimization procedure over multiple iterations and a memory of the cost function values. The different optimization algorithms, i.e. vanilla gradient descent, Adagrad, RMSProp and Adam, are implemented as subclasses that override the update rule for the parameters $\vec \theta$ in each iteration. The cost functions for OLS, Ridge and Lasso regression are also implemented as separate classes that provide methods to calculate the cost and its gradient. The stochastic gradient descent methods are implemented as further subclasses that modify the data used for fitting in each iteration. In the stochastic gradient descent method, the cost history property is furthermore modified to return the per epoch average of the cost function values. The resampling methods, i.e. bootstrapping and k-fold cross-validation, are implemented as methods with identical interfaces which provide multiple sets of training and test data. This allows to easily switch between the different resampling methods when evaluating the performance of a model.
report/chapters/results.tex
+76
View File
@@ -0,0 +1,76 @@
\subsection{Influence of the Number of Parameters on Model Performance}
\label{sec:results_num_params}
\subsubsection{Ordinary Least Squares}
\label{sec:results_ols}
Even the best optimization algorithm cannot turn a bad model into a good one. If a model is not suited to describe the trend in the data irrelevant of the parameters, it is logical to assume that no optimization algorithm can find a good fit. A common approach to counteract this problem is to increase the number of parameters in a model and use a very flexible model, e.g. a polynomial of high degree. In this section the influence of the number of parameters on the model performance is studied in the context of ordinary least squares (OLS) regression.
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{../figures/ols_mse_r2.pdf}
\caption{Mean squared error (MSE) and $R^2$ score for polynomial fits of different degrees to noisy data from the Runge function using ordinary least squares regression. $\N=\num{80000}, \Nt = \num{20000}$.}
\label{fig:ols_mse_r2}
\end{figure}
How the mean squared error (MSE) and the $R^2$ score depend on the polynomial degree of the model is shown in \cref{fig:ols_mse_r2}. As expected, the MSE decreases and the $R^2$ score increases with increasing polynomial degree. However, it is also visible that for very high polynomial degrees the performance does not improve significantly anymore. Using a dataset with \num{80000} samples, no negative side effects of overfitting are visible even for polynomial degrees as high as 20. To highlight the danger of negative impact due to high parameter count it is also interesting to study the paramter values of the fitted models. \Cref{fig:ols_params} shows the parameter values for polynomial fits of different degrees to the same dataset. It is visible that for high polynomial degrees the parameter values become very large. Especially it is observable that the parameter values flip sign for every second parameter, i.e. compensating negative effects of one parameter with the next. While such behavior is not necessarily a problem within the feature space of the training data, it can lead to very bad performance outside of this feature space. This is especially relevant for extrapolation tasks, where the model is used to predict values outside of the feature space of the training data. In such cases, the model can produce completely meaningless results.
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{../figures/ols_parameter_plot.pdf}
\caption{Parameter values for polynomial fits of different degrees to noisy data from the Runge function using ordinary least squares regression. $\N=\num{80000}$.}
\label{fig:ols_params}
\end{figure}
In the limit of small datasets this issue becomes even more relevant. As the number of parameters approaches the number of data points, the model can fit the data perfectly, leading to a MSE of zero. However, this is usually not a desirable outcome, as the model will not generalize well to new data, as it has simply \textquote{memorized} the noise in the training data. This is a classic example of overfitting, where the model performs well on the training data but poorly on unseen data. In such cases, it is crucial to use techniques to prevent overfitting, such as regularization or cross-validation.
\subsubsection{Regularization Techniques}
\label{sec:results_reg_techniques}
To counteract the negative effects of high parameter count, regularization techniques such as Ridge and Lasso regression can be used. These techniques add a penalty term to the cost function that discourages large parameter values. In Ridge regression, the penalty term is proportional to the square of the parameter values, while in Lasso regression, it is proportional to the absolute value of the parameter values. This leads to a trade-off between fitting the data well and keeping the parameter values small.
With an increasing regularization strength $\lambda$ the parameter values are pushed towards zero, as visible in \cref{fig:ridge_parameter_plot}. This effect is especially strong for $\lambda > 0$. For very high regularization strengths the parameter values become very small, leading to a model that is almost constant. This is also visible in \cref{fig:ridge_mse_heatmap}, where the MSE is shown for polynomial fits of different degrees to noisy data from the Runge function using Ridge regression with different regularization strengths $\lambda$. For very high regularization strengths the MSE increases significantly, as the model is not flexible enough to fit the data well anymore. The optimal regularization strength is promoted to a hyperparameter that needs to be tuned to the problem at hand. However, with a suitable choice of $\lambda$ it is possible to use models with a high number of parameters without suffering from the negative effects of overfitting.
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{../figures/ridge_parameter_plot.pdf}
\caption{Parameter values for polynomial fits of different degrees to noisy data from the Runge function using Ridge regression with different regularization strengths $\lambda$. $\N=\num{80000}$.}
\label{fig:ridge_parameter_plot}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{../figures/ridge_mse_heatmap.pdf}
\caption{Mean squared error (MSE) for polynomial fits of different degrees to noisy data from the Runge function using Ridge regression with different regularization strengths $\lambda$. $\N=\num{80000}, \Nt = \num{20000}$.}
\label{fig:ridge_mse_heatmap}
\end{figure}
\subsection{Stochastic Gradient Descent}
\label{sec:results_sgd}
While the high variance of models with a high number of parameters can be counteracted with a large amount of data, the computational cost of fitting such models can become very high. This is especially relevant in the context of big data, where datasets can contain millions or even billions of samples. In such cases, it can be infeasible to use optimization algorithms that require the computation of the gradient over the entire dataset, such as ordinary gradient descent. For a dataset with $\N$ samples and a model with $p$ parameters, the computational cost of calculating the gradient
\begin{equation}
\nabla_\theta \cols = \frac{2}{N}\left( X^T X \vec \theta - X^T \vec{y} \right),
\end{equation}
is of order $\mathcal{O}(N p^2)$, as the matrix-vector products $X^T X$ needs to be computed. For very large datasets this can lead to very long computation times, as the entire dataset needs to be processed to compute the gradient. A common approach to counteract this problem is to use stochastic gradient descent (SGD) or mini-batch gradient descent, where the gradient is computed over a small subset of the data, called a batch. This leads to a significant reduction in computational cost, as the gradient can be computed in $\mathcal{O}(B p^2)$, where $B$ is the batch size. However, this comes at the cost of increased variance in the gradient estimate, as the gradient is only computed over a small subset of the data. This can lead to slower convergence and more oscillations in the optimization process.
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{../figures/optimization_performance.pdf}
\caption{Convergence of GD and SGD for polynomial fits of degree 10 to noisy data from the Runge function using OLS and Ridge regression with $\lambda = \num{0.1}$. $\N=\num{800000}, B=\num{512}, N_\mathrm{batches} = \num{100}$ and $\sigma = \num{0.1}$.}
\label{fig:optimization_performance}
\end{figure}
\Cref{fig:optimization_performance} shows the large decrease in computational cost, i.e. the decrease in computation time for \num{1000} epochs when using SGD instead of GD. While the decrease in computation time is significant, the result is even less black and white, when taking a closer look at the details.
While conventional GD can benefit largely by precomputing the matrix products $X^T X$ and $X^T \vec{y}$, this is not possible for SGD, as the batches change in every epoch. This leads to a significant increase in computation time per epoch for SGD compared to GD. However, as the number of epochs needed to reach convergence is usually much lower for SGD than for GD, the overall computation time can still be significantly lower for SGD. This is especially relevant for very large datasets, where the computational cost of GD can become prohibitive. Because while GD needs to process the entire dataset and thus the computational cost is of order $\mathcal{O}(N p^2)$, SGD only needs to process a small batch of size $B$ and thus the computational cost is of order $\mathcal{O}(B p^2)$ and invariant of the size of the dataset, given the batch size is large enough to estimate the complete gradient well enough. This leads to a significant reduction in computation time for very large datasets, as the computational cost of SGD does not increase with the size of the dataset.
A second aspect of SGD is visible in \cref{fig:optimization_performance}. While GD converges smoothly to the optimal solution, SGD shows significant oscillations around the optimal solution. This is a direct consequence of the increased variance in the gradient estimate, as the gradient is only computed over a small subset of the data. This can lead to slower convergence and even no convergence at all, if the batch size is too small or the noise in the data is too high. The intrinsic noise in the gradient estimate leads also to the issue, that SGD may be unable to reach an optimal solution with a large amount of noise in the training data. To counteract this problem, the noise in the dataset has been reduced by one order of magnitude compared to the previous sections. However, even with a reduced noise level, SGD is unable to reach the optimal solution. Further fine tuning of the new hyperparameters introduced by SGD, i.e. the batch size and the batches per epoch, could lead to better results. However, this is a non-trivial task and is highly dependent on the problem at hand. To arrive at a robust solution it might be necessary to increase the batch size until reaching the limit of full-batch GD, which would defeat the purpose of using SGD in the first place.
\subsection{Alternative Optimization Algorithms}
\label{sec:results_alt_opt_algos}
\subsection{Bias-Variance Tradeoff and Resampling Techniques}
\label{sec:results_bias_variance}
report/main.pdf
BIN
View File
Binary file not shown.
report/main.tex
+20 -9
View File
@@ -16,14 +16,16 @@
\usepackage{booktabs}
\usepackage{multirow,bigdelim}
\usepackage{float}
\usepackage{todonotes}
\usepackage{upgreek} %upalpha in Saxena2021 Reference
\usepackage[utf8]{inputenc}
\usepackage{hyperref}
\hypersetup{breaklinks=true,colorlinks=true,linkcolor=blue,citecolor=blue,filecolor=magenta,urlcolor=blue}
\usepackage{siunitx}
\usepackage{cleveref}
\usepackage{csquotes}
\usepackage{xcolor}
@@ -31,6 +33,16 @@
%\newcommand{\comment}[1]{\textcolor{blue}{#1}}
%\newcommand{\WN}[1]{{\color{red} #1}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\cols}{C_\mathrm{OLS}}
\newcommand{\colr}{C_\mathrm{Ridge}}
\newcommand{\coll}{C_\mathrm{Lasso}}
\newcommand{\mse}{\mathrm{MSE}}
\newcommand{\bias}{\mathrm{Bias}}
\newcommand{\var}{\mathrm{Var}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\N}{N_\mathrm{train}}
\newcommand{\Nt}{N_{\mathrm{test}}}
\makeatletter
\def\@bibdataout@aps{%
@@ -72,22 +84,21 @@ apsrev41Control%
\tableofcontents
\section{Introduction}
\section{Introduction} \label{sec:introduction}
\input{chapters/introduction.tex}
\section{Methods}
\section{Methods} \label{sec:methods}
\input{chapters/methods.tex}
\section{Results}
\section{Results} \label{sec:results}
\input{chapters/results.tex}
\section{Conclusion}
\section{Conclusion} \label{sec:conclusion}
\input{chapters/conclusion.tex}
\section{Perspective}
\input{chapters/perspective.tex}
\section{Perspective} \label{sec:perspective}
\input{chapters/perspective.tex}
\bibliography{include/sources/references} % add references to this file
\end{document}