\subsection{Influence of the Number of Parameters on Model Performance}
\label{sec:results_num_params}
\subsubsection{Ordinary Least Squares}
\label{sec:results_ols}
Even the best optimization algorithm cannot turn a bad model into a good one. If a model is not suited to describe the trend in the data irrelevant of the parameters, it is logical to assume that no optimization algorithm can find a good fit. A common approach to counteract this problem is to increase the number of parameters in a model and use a very flexible model, e.g. a polynomial of high degree. In this section the influence of the number of parameters on the model performance is studied in the context of ordinary least squares (OLS) regression.

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{../figures/ols_mse_r2.pdf}
    \caption{Mean squared error (MSE) and $R^2$ score for polynomial fits of different degrees to noisy data from the Runge function using ordinary least squares regression. $\N=\num{80000}, \Nt = \num{20000}$.}
    \label{fig:ols_mse_r2}
\end{figure}

How the mean squared error (MSE) and the $R^2$ score depend on the polynomial degree of the model is shown in \cref{fig:ols_mse_r2}. As expected, the MSE decreases and the $R^2$ score increases with increasing polynomial degree. However, it is also visible that for very high polynomial degrees the performance does not improve significantly anymore. Using a dataset with \num{80000} samples, no negative side effects of overfitting are visible even for polynomial degrees as high as 20. To highlight the danger of negative impact due to high parameter count it is also interesting to study the paramter values of the fitted models. \Cref{fig:ols_params} shows the parameter values for polynomial fits of different degrees to the same dataset. It is visible that for high polynomial degrees the parameter values become very large. Especially it is observable that the parameter values flip sign for every second parameter, i.e. compensating negative effects of one parameter with the next. While such behavior is not necessarily a problem within the feature space of the training data, it can lead to very bad performance outside of this feature space. This is especially relevant for extrapolation tasks, where the model is used to predict values outside of the feature space of the training data. In such cases, the model can produce completely meaningless results.

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{../figures/ols_parameter_plot.pdf}
    \caption{Parameter values for polynomial fits of different degrees to noisy data from the Runge function using ordinary least squares regression. $\N=\num{80000}$.}
    \label{fig:ols_params}
\end{figure}

In the limit of small datasets this issue becomes even more relevant. As the number of parameters approaches the number of data points, the model can fit the data perfectly, leading to a MSE of zero. However, this is usually not a desirable outcome, as the model will not generalize well to new data, as it has simply \textquote{memorized} the noise in the training data. This is a classic example of overfitting, where the model performs well on the training data but poorly on unseen data. In such cases, it is crucial to use techniques to prevent overfitting, such as regularization or cross-validation.

\subsubsection{Regularization Techniques}
\label{sec:results_reg_techniques}
To counteract the negative effects of high parameter count, regularization techniques such as Ridge and Lasso regression can be used. These techniques add a penalty term to the cost function that discourages large parameter values. In Ridge regression, the penalty term is proportional to the square of the parameter values, while in Lasso regression, it is proportional to the absolute value of the parameter values. This leads to a trade-off between fitting the data well and keeping the parameter values small.
With an increasing regularization strength $\lambda$ the parameter values are pushed towards zero, as visible in \cref{fig:ridge_parameter_plot}. This effect is especially strong for $\lambda > 0$. For very high regularization strengths the parameter values become very small, leading to a model that is almost constant. This is also visible in \cref{fig:ridge_mse_heatmap}, where the MSE is shown for polynomial fits of different degrees to noisy data from the Runge function using Ridge regression with different regularization strengths $\lambda$. For very high regularization strengths the MSE increases significantly, as the model is not flexible enough to fit the data well anymore. The optimal regularization strength is promoted to a hyperparameter that needs to be tuned to the problem at hand. However, with a suitable choice of $\lambda$ it is possible to use models with a high number of parameters without suffering from the negative effects of overfitting.

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{../figures/ridge_parameter_plot.pdf}
    \caption{Parameter values for polynomial fits of different degrees to noisy data from the Runge function using Ridge regression with different regularization strengths $\lambda$. $\N=\num{80000}$.}
    \label{fig:ridge_parameter_plot}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{../figures/ridge_mse_heatmap.pdf}
    \caption{Mean squared error (MSE) for polynomial fits of different degrees to noisy data from the Runge function using Ridge regression with different regularization strengths $\lambda$. $\N=\num{80000}, \Nt = \num{20000}$.}
    \label{fig:ridge_mse_heatmap}
\end{figure}





\subsection{Stochastic Gradient Descent}
\label{sec:results_sgd}

While the high variance of models with a high number of parameters can be counteracted with a large amount of data, the computational cost of fitting such models can become very high. This is especially relevant in the context of big data, where datasets can contain millions or even billions of samples. In such cases, it can be infeasible to use optimization algorithms that require the computation of the gradient over the entire dataset, such as ordinary gradient descent. For a dataset with $\N$ samples and a model with $p$ parameters, the computational cost of calculating the gradient
\begin{equation}
    \nabla_\theta \cols = \frac{2}{N}\left( X^T X \vec \theta - X^T \vec{y} \right),
\end{equation}
is of order $\mathcal{O}(N p^2)$, as the matrix-vector products $X^T X$ needs to be computed. For very large datasets this can lead to very long computation times, as the entire dataset needs to be processed to compute the gradient. A common approach to counteract this problem is to use stochastic gradient descent (SGD) or mini-batch gradient descent, where the gradient is computed over a small subset of the data, called a batch. This leads to a significant reduction in computational cost, as the gradient can be computed in $\mathcal{O}(B p^2)$, where $B$ is the batch size. However, this comes at the cost of increased variance in the gradient estimate, as the gradient is only computed over a small subset of the data. This can lead to slower convergence and more oscillations in the optimization process. 

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{../figures/optimization_performance.pdf}
    \caption{Convergence of GD and SGD for polynomial fits of degree 10 to noisy data from the Runge function using OLS and Ridge regression with $\lambda = \num{0.1}$. $\N=\num{800000}, B=\num{512}, N_\mathrm{batches} = \num{100}$ and $\sigma = \num{0.1}$.}
    \label{fig:optimization_performance}
\end{figure}

\Cref{fig:optimization_performance} shows the large decrease in computational cost, i.e. the decrease in computation time for \num{1000} epochs when using SGD instead of GD. While the decrease in computation time is significant, the result is even less black and white, when taking a closer look at the details.
While conventional GD can benefit largely by precomputing the matrix products $X^T X$ and $X^T \vec{y}$, this is not possible for SGD, as the batches change in every epoch. This leads to a significant increase in computation time per epoch for SGD compared to GD. However, as the number of epochs needed to reach convergence is usually much lower for SGD than for GD, the overall computation time can still be significantly lower for SGD. This is especially relevant for very large datasets, where the computational cost of GD can become prohibitive. Because while GD needs to process the entire dataset and thus the computational cost is of order $\mathcal{O}(N p^2)$, SGD only needs to process a small batch of size $B$ and thus the computational cost is of order $\mathcal{O}(B p^2)$ and invariant of the size of the dataset, given the batch size is large enough to estimate the complete gradient well enough. This leads to a significant reduction in computation time for very large datasets, as the computational cost of SGD does not increase with the size of the dataset.

A second aspect of SGD is visible in \cref{fig:optimization_performance}. While GD converges smoothly to the optimal solution, SGD shows significant oscillations around the optimal solution. This is a direct consequence of the increased variance in the gradient estimate, as the gradient is only computed over a small subset of the data. This can lead to slower convergence and even no convergence at all, if the batch size is too small or the noise in the data is too high. The intrinsic noise in the gradient estimate leads also to the issue, that SGD may be unable to reach an optimal solution with a large amount of noise in the training data. To counteract this problem, the noise in the dataset has been reduced by one order of magnitude compared to the previous sections. However, even with a reduced noise level, SGD is unable to reach the optimal solution. Further fine tuning of the new hyperparameters introduced by SGD, i.e. the batch size and the batches per epoch, could lead to better results. However, this is a non-trivial task and is highly dependent on the problem at hand. To arrive at a robust solution it might be necessary to increase the batch size until reaching the limit of full-batch GD, which would defeat the purpose of using SGD in the first place.



\subsection{Alternative Optimization Algorithms}
\label{sec:results_alt_opt_algos}


\subsection{Bias-Variance Tradeoff and Resampling Techniques}
\label{sec:results_bias_variance}