Runge comparison wo noise

report/main.tex

@@ -281,6 +281,13 @@ As we mentioned before, there is the possibility for our model to overadjust for
 A further hindrance in our approach with using the output of a regression model to make classifications is, that for an efficient workflow, we need a full but independent prediction of the dataset. To achieve this, we use a workflow inspired by cross-validation. In cross-validation you use multiple folds of the dataset, whereby for every fold, a different part of the dataset is used as testing data, and the rest of the dataset is used for training purposes. This provides independence of training and testing data while being able to test the models general performance for the entire dataset. For out-of-fold prediction, we use the same approach of changing the training data to leave out a specific part each time, but furthermore we use the model trained on each training set, to make predictions for the remaining set. Combining all the predicted sets into one set, allows for having independent predictions for the entire dataset. Having the predictions be independent is important, as otherwise no statement can be made on the generalization ability of the workflow. Using this approach we can expect the same performance on new and yet unseen data.
 
 \subsection{Implementation}
+\begin{figure}
+  \centering
+  \includegraphics[width=\columnwidth]{../notebooks/runge_no_noise_comparison.pdf}
+  \caption{Comparison of predictions made by OLS regression of polynomial degree 10 and a FFNN with two hidden layers of 100 nodes each on the Runge function without noise.}
+  \label{fig:runge_comparison}
+\end{figure}
+
 All the code necessary to use a FFNN is implemented from scratch in \texttt{Python3} as the \texttt{easynn} library. The source code therefore is located in the \texttt{/src/easynn/} folder of the accompanying git repository \footnote{\url{https://github.uio.no/larsbog/FYSSTK-Project2}}. It is separated into two main modules, called \texttt{schedulers} and \texttt{feedforward}. The scheduler modules use modified versions of our previous work on using gradient descent techniques for ordinary least squares techniques. It is also structured in a modular way, so that scheduling tasks like early stopping and dynamic learning rate adjustment are easy to implement in further versions. The base scheduler implements an update method of signature \texttt{def update(self, gradients: gradient\_\-type) -> gradient\_\-type:}, with \texttt{gradient\_\-type = List[\-Tuple[\-np.ndarray, np.ndarray]]
 }. This can be overwritten to implement any optimization algorithm. The method takes the gradients of the cost functions with respect to the weights and biases of each layer. Each layer is assigned a tuple of two arrays of the gradients. It should then return the updates for each of the weights and biases in the same format. The training process which fetches new updates after each evaluation of the loss and gradients will continue will the \texttt{cont} property of the base class is true.
 
@@ -300,7 +307,8 @@ As a further validation of the functionality, we use the Runge function
 \begin{equation}
 	f(x) = \frac{1}{1 + 25 x^2},
 \end{equation}
-with added Gaussian noise of standard deviation \num{1.0} to create a regression dataset. On this dataset of size $N = \num{100000}$ and \qty{80}{\percent}-\qty{20}{\percent} train-test-splitting we achieve a MSE of \numrange{0.9136}{0.9178}, using between one and two hidden layers with either 50 or 100 nodes per hidden layer. This exceeds the performance of our previous work on least squares regression using different regularization and optimization techniques \cite{bognerRegularizationOptimizationAll2025}, where a MSE of \num{0.92} was achieved using the same dataset \footnote{The $x$ values were expanded into multiple of features containing polynomial multiples $x^n$ of themselves. For optimal performance $n \in \{0, 1, \dots, \geq 10\}$ were required.} and train-test-splitting. This motivates further the use of neural networks for regression tasks, as possible non-linearities in the data can be modeled more easily. Least squares methods on the other hand provide only linear combinations of the input features.
+with added Gaussian noise of standard deviation \num{1.0} to create a regression dataset. On this dataset of size $N = \num{100000}$ and \qty{80}{\percent}-\qty{20}{\percent} train-test-splitting we achieve a MSE of \numrange{0.9136}{0.9178}, using between one and two hidden layers with either 50 or 100 nodes per hidden layer. This exceeds the performance of our previous work on least squares regression using different regularization and optimization techniques \cite{bognerRegularizationOptimizationAll2025}, where a MSE of \num{0.92} was achieved using the same dataset \footnote{The $x$ values were expanded into multiple of features containing polynomial multiples $x^n$ of themselves. For optimal performance $n \in \{0, 1, \dots, \geq 10\}$ were required.} and train-test-splitting. 
+In the case of no added noise, the difference is even more stark. In the case of 2 hidden layers with Leaky ReLU activation and 100 nodes per hidden layer on $N= \num{1000}$ points, we achieve $\mathrm{MSE} = \num{1.037e-5}$. OLS methods achieve a performance of \num{0.01984} in this case. \Cref{fig:runge_comparison} shows the predictions by both methods in form of a plot. This motivates further the use of neural networks for regression tasks, as possible non-linearities in the data can be modeled more easily. Least squares methods on the other hand provide only linear combinations of the input features.
 
 
 \subsection{Use of AI tools}