Optimizing numerical models to model a set of data has been a challenge of the natural sciences for many decades. But with the introduction of artificial neural networks in all disciplines over the last years a special focus on those optimization techniques is relevant.

In this paper different methods of such optimization are studied in the context of large datasets and big data with possibly noise-affected input data. To render the best possible fits a multitude of cost functions, minimization algorithms and other approaches to reduce the computation associated with the optimization process are studied and evaluated. Reducing the computational cost is also important to improve the economical and ecological footprint of training large models on big data, as the use of data-driven methods in all areas of life is increasing rapidly. To measure the possible improvements with state-of-the-art methods, the duration of optimization and the quality of the fit are evaluated. Assuming constant power consumption of the hardware, a reduction in computation time directly translates to a reduction in energy consumption and thus a smaller ecological and economical footprint.

In a first step, the dependence of the bias-variance tradeoff on the number of degrees of freedom in a model is studied. For this purpose, polynomial models of different degrees are fitted to data generated from the Runge function with significant noise. The results are then compared to different Ridge and Lasso models to study the effect of regularization on the bias-variance tradeoff. In the context of big data, the performance of different numerical minimization algorithms is studied. For this purpose, the performance of gradient descent, stochastic gradient descent and different optimization algorithms such as Adam are compared in the context of Ridge and Lasso regression. Furthermore, the possible advantages and disadvantages of stochastic gradient descent techniques are evaluated. Finally, the effect of resampling methods such as bootstrapping and k-fold cross-validation on the bias-variance tradeoff is studied.

In the following section the theoretical background of the methods used in this paper is presented. In \cref{sec:methods} the implementation of these methods is discussed. The results of applying these methods to different datasets are presented in \cref{sec:results}. Finally, a conclusion is drawn in \cref{sec:conclusion} and an outlook on possible future work is given in \cref{sec:perspective}.
