Hopefully I'm done

report/chapters/conclusion.tex

@@ -1 +1 @@
-In summary it was shown that in the context of big data and complex models to describe relations in the data there are many challenges to overcome. The choice of model, on what metric to optimize the model -- i.e. the cost function -- and how to perform the optimization are all crucial choices that can have a significant impact on the performance of the model. The use of regularization techniques can help to prevent overfitting and improve the generalization performance of the model. The number of degrees of freedom in the model directly influcences the bias-variance trade-off and only with a good balance between the two can a model perform well on unseen data. Finally, the choice of optimization algorithm can have a significant impact on the computational cost of the models training process. Using good optimization algorithms can help to speed up the convergence of the models parameters. Combining this with other innovative techniques such as stochastic gradient descent can help to further reduce the computational cost of training complex models on large datasets.
+It was shown that in the context of big data and complex models to describe relations in the data there are many challenges to overcome. The choice of model, on what metric to optimize the model -- i.e. the cost function -- and how to perform the optimization are all crucial choices that can have a significant impact on the performance of the model. The use of regularization techniques can help to prevent overfitting and improve the generalization performance of the model. The number of degrees of freedom in the model directly influcences the bias-variance trade-off and only with a good balance between the two can a model perform well on unseen data. Finally, the choice of optimization algorithm can have a significant impact on the computational cost of the models training process. Using good optimization algorithms can help to speed up the convergence of the models parameters. Combining this with other innovative techniques such as stochastic gradient descent can help to further reduce the computational cost of training complex models on large datasets.

report/chapters/introduction.tex

@@ -1,8 +1,7 @@
 Optimizing numerical models to model a set of data has been a challenge of the natural sciences for many decades. But with the introduction of artificial neural networks in all disciplines over the last years a special focus on those optimization techniques is relevant.
 
-In this paper different methods of such optimization are studied in the context of large datasets and big data with possibly noise-affected input data. To render the best possible fits a multitude of cost functions, minimization algorithms and other approaches to reduce the computation associated with the optimization process are studied and evaluated. Reducing the computational cost is also important due improve the economical and ecological footprint of training large models on big data, as the spread of datadriven methods in all areas of life is increasing rapidly.
+In this paper different methods of such optimization are studied in the context of large datasets and big data with possibly noise-affected input data. To render the best possible fits a multitude of cost functions, minimization algorithms and other approaches to reduce the computation associated with the optimization process are studied and evaluated. Reducing the computational cost is also important to improve the economical and ecological footprint of training large models on big data, as the use of datadriven methods in all areas of life is increasing rapidly. To measure the possible improvements with state-of-the-art methods, the duration of optimization and the quality of the fit are evaluated. Assuming constant power consumption of the hardware, a reduction in computation time directly translates to a reduction in energy consumption and thus a smaller ecological and economical footprint.
-
-In the following section the theoretical background of the methods used in this paper is presented. In \cref{sec:methods} secondly the implementation of these methods is discussed. The results of applying these methods to different datasets are presented in \cref{sec:results}. Finally, a conclusion is drawn in \cref{sec:conclusion} and an outlook on possible future work is given in \cref{sec:perspective}.
 
 In a first step, the dependence of the bias-variance tradeoff on the number of degrees of freedom in a model is studied. For this purpose, polynomial models of different degrees are fitted to data generated from the Runge function with significant noise. The results are then compared to different Ridge and Lasso models to study the effect of regularization on the bias-variance tradeoff. In the context of big data, the performance of different numerical minimization algorithms is studied. For this purpose, the performance of gradient descent, stochastic gradient descent and different optimization algorithms such as Adam are compared in the context of Ridge and Lasso regression. Furthermore the possible advantages and disadvantages of stochastic gradient descent techniques are evaluated. Finally, the effect of resampling methods such as bootstrapping and k-fold cross-validation on the bias-variance tradeoff is studied.
 
+In the following section the theoretical background of the methods used in this paper is presented. In \cref{sec:methods} the implementation of these methods is discussed. The results of applying these methods to different datasets are presented in \cref{sec:results}. Finally, a conclusion is drawn in \cref{sec:conclusion} and an outlook on possible future work is given in \cref{sec:perspective}.

report/chapters/results.tex

@@ -11,7 +11,7 @@ Even the best optimization algorithm cannot turn a bad model into a good one. If
     \label{fig:ols_mse_r2}
 \end{figure}
 
-How the mean squared error (MSE) and the $R^2$ score depend on the polynomial degree of the model is shown in \cref{fig:ols_mse_r2}. As expected, the MSE decreases and the $R^2$ score increases with increasing polynomial degree. However, it is also visible that for very high polynomial degrees the performance does not improve significantly anymore. Using a dataset with \num{80000} samples, no negative side effects of overfitting are visible even for polynomial degrees as high as 20. To highlight the danger of negative impact due to high parameter count it is also interesting to study the paramter values of the fitted models. \Cref{fig:ols_params} shows the parameter values for polynomial fits of different degrees to the same dataset. It is visible that for high polynomial degrees the parameter values become very large. Especially it is observable that the parameter values flip sign for every second parameter, i.e. compensating negative effects of one parameter with the next. While such behavior is not necessarily a problem within the feature space of the training data, it can lead to very bad performance outside of this feature space. This is especially relevant for extrapolation tasks, where the model is used to predict values outside of the feature space of the training data. In such cases, the model can produce completely meaningless results.
+How the mean squared error (MSE) and the $R^2$ score depend on the polynomial degree of the model is shown in \cref{fig:ols_mse_r2}. As expected, the MSE decreases and the $R^2$ score increases with increasing polynomial degree. However, it is also visible that for very high polynomial degrees the performance does no longer improve significantly. Using a dataset with \num{80000} samples, no negative side effects of overfitting are visible even for polynomial degrees as high as 20. To highlight the danger of negative impact due to high parameter count it is also interesting to study the parameter values of the fitted models. \Cref{fig:ols_params} shows the parameter values for polynomial fits of different degrees to the same dataset. It is visible that for high polynomial degrees the parameter values become very large, reaching values of up to \num{+-e4}. Especially it is observable that the parameter values flip sign for every second parameter, i.e. compensating negative effects of one parameter with the next. While such behavior is not necessarily a problem within the feature space of the training data, it can lead to bad performance outside of this feature space, i.e. performance that is worse than the most simple approximation of constant output values. This is especially relevant for extrapolation tasks, where the model is used to predict values outside of the feature space of the training data. In such cases, the model can produce completely meaningless results.
 
 \begin{figure}
     \centering
@@ -25,7 +25,7 @@ In the limit of small datasets this issue becomes even more relevant. As the num
 \subsubsection{Regularization Techniques}
 \label{sec:results_reg_techniques}
 To counteract the negative effects of high parameter count, regularization techniques such as Ridge and Lasso regression can be used. These techniques add a penalty term to the cost function that discourages large parameter values. In Ridge regression, the penalty term is proportional to the square of the parameter values, while in Lasso regression, it is proportional to the absolute value of the parameter values. This leads to a trade-off between fitting the data well and keeping the parameter values small.
-With an increasing regularization strength $\lambda$ the parameter values are pushed towards zero, as visible in \cref{fig:ridge_parameter_plot}. This effect is especially strong for $\lambda > 0$. For very high regularization strengths the parameter values become very small, leading to a model that is almost constant. This is also visible in \cref{fig:ridge_mse_heatmap}, where the MSE is shown for polynomial fits of different degrees to noisy data from the Runge function using Ridge regression with different regularization strengths $\lambda$. For very high regularization strengths the MSE increases significantly, as the model is not flexible enough to fit the data well anymore. The optimal regularization strength is promoted to a hyperparameter that needs to be tuned to the problem at hand. However, with a suitable choice of $\lambda$ it is possible to use models with a high number of parameters without suffering from the negative effects of overfitting.
+With an increasing regularization strength $\lambda$ the parameter values are pushed towards zero, as visible in \cref{fig:ridge_parameter_plot}. This effect is especially strong for $\lambda > 0$. For very high regularization strengths the parameter values become very small, leading to a model that is almost constant. This is also visible in \cref{fig:ridge_mse_heatmap}, where the MSE is shown for polynomial fits of different degrees to noisy data from the Runge function using Ridge regression with different regularization strengths $\lambda$. For very high regularization strengths the MSE increases significantly, as the model is not flexible enough to fit the data well. The optimal regularization strength is promoted to a hyperparameter that needs to be tuned to the problem at hand. However, with a suitable choice of $\lambda$ it is possible to use models with a high number of parameters without suffering from the negative effects of overfitting.
 
 \begin{figure}
     \centering
@@ -99,7 +99,7 @@ A different approach to limit the computational cost of fitting models with a hi
 
 To evaluate the bias-variance tradeoff of the optimized models, the MSE is decomposed into its bias and variance components as introduced in \cref{sec:bias_variance_tradeoff}. The results are shown in \cref{fig:bias_variance_tradeoff}. To accurately depict the bias-variance tradeoff, there were two sets of test data used. To compute the bias, no noise was added to the test data, while for the rest of the computations the same noise level as in the training data was used. It is visible that for increasing polynomial degrees the variance increases, while the bias stays almost constant. This is a direct consequence of the increased flexibility of the model with increasing polynomial degree. To ensure numerical stability, Ridge regression with a very small regularization strength of $\lambda = \num{1e-10}$ has been used to compute the bias-variance decomposition in an approximation of OLS. This ensures that the matrix inversion in the computation of the model parameters is numerically stable, while the regularization term has almost no effect on the model performance.
 The optimal model complexity is usually found at the point where the sum of bias and variance is minimal. Usually this coincides with the point of the minimum of the mean squared error. How the MSE behaves on average for $k$-fold cross-validation and bootstrap resampling techniques is also shown in \cref{fig:bias_variance_tradeoff}.
-Both resampling techniques show a minimum in the MSE for the polynomial degree of 2. The crossvalidation technique shows a much more constant MSE value over the entire range of polynomial degrees, while the bootstrap technique shows a more pronounced minimum. This indicates that the bootstrap technique is more sensitive to the choice of polynomial degree and thus provides a better estimate in the case of hyperparameter tuning at the cost of a much higher computational cost.
+Both resampling techniques display a minimum in the MSE for the polynomial degree of 2. The crossvalidation technique presents a much more constant MSE value over the entire range of polynomial degrees, while the bootstrap technique expresses a more pronounced minimum. This indicates that the bootstrap technique is more sensitive to the choice of polynomial degree and thus provides a better estimate in the case of hyperparameter tuning at the cost of a much higher computational cost.
 
 This trend of optimal model complexity being achieved somewhere between the high bias case of low parameter count and the high variance case of high parameter count is a general property of machine learning models and is not limited to polynomial regression with OLS cost functions. As seen in \cref{fig:kfold_mse_comp} this trend of a minimum in the MSE is also observable for Ridge and Lasso regression. 
 

report/main.tex

@@ -1,43 +1,24 @@
+\documentclass[amsmath,amssymb,twocolumn,aps]{revtex4}
 
-\documentclass[aps,rmp,reprint,amsmath,amssymb,graphicx,longbibliography,twoside]{revtex4-1}
+% allows special characters (including æøå)
-
-\usepackage{bm}
-\usepackage{graphicx}
-\usepackage{epstopdf}
-\usepackage{wrapfig}
-\usepackage{array} 
-\usepackage{listings}
-\usepackage[para,online,flushleft]{threeparttablex}
-\usepackage{booktabs,dcolumn}
-\usepackage{color}
- 
-
-\usepackage{textpos}
-\usepackage{booktabs}
-\usepackage{multirow,bigdelim}
-\usepackage{float}
-\usepackage{todonotes}
-
-% tikz decorations
-\usepackage{tikz}
-\usetikzlibrary{decorations.pathreplacing}
-
-\usepackage{upgreek} %upalpha in Saxena2021 Reference
 \usepackage[utf8]{inputenc}
-\usepackage{hyperref}
+%\usepackage [norsk]{babel} %if you write norwegian
-\hypersetup{breaklinks=true,colorlinks=true,linkcolor=blue,citecolor=blue,filecolor=magenta,urlcolor=blue}
+\usepackage[english]{babel}  %if you write english
 
-\usepackage{siunitx}
+\usepackage{amssymb}  % mathematical symbols (physics imports amsmath)
+\usepackage{graphicx}         % include graphics such as plots
+\usepackage[table]{xcolor}
+\usepackage{xcolor}           % set colors
+\usepackage{hyperref}         % automagic cross-referencing 
+\usepackage{float}			  % force placement of tables and figures
 \usepackage{cleveref}
-\usepackage{csquotes}
+\usepackage{tikz}
 
+\usetikzlibrary{decorations.pathreplacing}
 
 \usepackage{xcolor}
 \definecolor{turq}{RGB}{64,224,208}
-%\newcommand{\contrib}[1]{\textcolor{red}{#1}}
+\usepackage{siunitx}
-%\newcommand{\comment}[1]{\textcolor{blue}{#1}}
-
-%\newcommand{\WN}[1]{{\color{red} #1}}
 \newcommand{\norm}[1]{\left\lVert#1\right\rVert}
 \newcommand{\cols}{C_\mathrm{OLS}}
 \newcommand{\colr}{C_\mathrm{Ridge}}
@@ -49,22 +30,6 @@
 \newcommand{\N}{N_\mathrm{train}}
 \newcommand{\Nt}{N_{\mathrm{test}}}
 
-\makeatletter
-\def\@bibdataout@aps{%
-\immediate\write\@bibdataout{%
-@CONTROL{%
-apsrev41Control%
-\longbibliography@sw{%
-    ,author="08",editor="1",pages="1",title="0",year="1"%
-    }{%
-    ,author="08",editor="1",pages="1",title="",year="1"%
-    }%
-  }%
-}%
-\if@filesw \immediate \write \@auxout {\string \citation {apsrev41Control}}\fi
-}
-\makeatother
-
 
 %\usepackage[left]{lineno}
 %\linenumbers
@@ -73,7 +38,7 @@ apsrev41Control%
 \begin{document}
 %\pagenumbering{gobble} 
 
-\title{Regression Analysis and Resampling Methods}
+\title{Regularization and Optimization is All You Need?}
 
 \author{Lars Bogner}
 \affiliation{\url{https://github.uio.no/larsbog/FYSSTK-Project1}}
@@ -81,6 +46,7 @@ apsrev41Control%
 
 \begin{abstract}
   The basis for many data-driven methods is the optimization of numerical models to describe relations in data. With the increasing availability of large datasets and the growing complexity of models, there is an increasing need for efficient optimization techniques that can handle the computational challenges associated with big data. Furthermore, a good optimization metric can be crucial to find the best possible model to describe the data. In this paper different methods of such optimization are studied in the context of large datasets and big data with possibly noise-affected input data. To render the best possible fits a multitude of cost functions, minimization algorithms and other approaches to reduce the computation associated with the optimization process are studied and evaluated. Reducing the computational cost is also important to improve the economical and ecological footprint of training large models on big data, as the spread of datadriven methods in all areas of life is increasing rapidly.
+  Using optimal optimization techniques the convergence can be improved by up to one order of magnitude.
 \end{abstract}
 
 \maketitle