Finalize rport

report/Exported Items.bib

@@ -143,6 +143,14 @@
   pages = {2825--2830}
 }
 
+@misc{ProductDataSheet2012,
+  title = {Product {{Data Sheet}} - {{DuraSoy One}}},
+  year = 2012,
+  month = mar,
+  publisher = {Eco Safety Products},
+  langid = {english}
+}
+
 @inproceedings{streetNuclearFeatureExtraction1993,
   title = {Nuclear Feature Extraction for Breast Tumor Diagnosis},
   booktitle = {{{IS}}\&{{T}}/{{SPIE}}'s {{Symposium}} on {{Electronic Imaging}}: {{Science}} and {{Technology}}},

report/biblio.bib

@@ -143,6 +143,14 @@
   pages = {2825--2830}
 }
 
+@misc{ProductDataSheet2012,
+  title = {Product {{Data Sheet}} - {{DuraSoy One}}},
+  year = 2012,
+  month = mar,
+  publisher = {Eco Safety Products},
+  langid = {english}
+}
+
 @inproceedings{streetNuclearFeatureExtraction1993,
   title = {Nuclear Feature Extraction for Breast Tumor Diagnosis},
   booktitle = {{{IS}}\&{{T}}/{{SPIE}}'s {{Symposium}} on {{Electronic Imaging}}: {{Science}} and {{Technology}}},

report/main.tex

@@ -64,7 +64,7 @@
 \tableofcontents
 
 \section{Introduction}
-Sampling the cell nuclei of tissue extracted using a fine needle aspirate from the breast allows for a very easy separation between benign and malignant tissue\cite{streetNuclearFeatureExtraction1993}. While very good performance is already shown with straightforward approaches, like decision trees \cite{bennettDecisionTreeConstruction1992}, we try to improve the efficiency of the detection process in this paper.
+Sampling the cell nuclei of tissue extracted using a fine needle aspirate from the breast allows for a very easy separation between benign and malignant tissue \cite{streetNuclearFeatureExtraction1993}. While very good performance is already shown with straightforward approaches, like decision trees \cite{bennettDecisionTreeConstruction1992}, we try to improve the efficiency of the detection process in this paper.
 
 Using a very small subset of features which can be obtained using basic image recognition, we enhance them to a larger set of nuclei features using a regression model, based on which the straightforward classification can be continued using the large set of features.
 
@@ -126,13 +126,13 @@ Starting from the features of radius and area from the Wisconsin Breast Cancer D
 
 Neural networks are the continuation of a linear combination by introducing nonlinearity. While linear regression uses a linear combination of input features to create an output, neural networks add a nonlinear function to the output of the linear combination. This is the entire secret behind the powerful possibilities neural networks have shown over the last years \cite{elstnerLectureMachineLearning2025,goodfellowDeepLearning2016}.
 
-A feedforward neural network (FFNN) uses a layered structure as shown in \cref{fig:neural_network}, where each node represents a separate linear combination of it's inputs. A general node uses the inputs $x_i$ to create a weighted sum $z$, which is then activated using a so called activation function $g(z)$. More detail on the activation function is provided in \cref{subsec:activation}. The output of the general node can then be calculated as
+A feedforward neural network (FFNN) uses a layered structure as shown in \cref{fig:neural_network}, where each node represents a separate linear combination of its inputs. A general node uses the inputs $x_i$ to create a weighted sum $z$, which is then activated using a so called activation function $g(z)$. More detail on the activation function is provided in \cref{subsec:activation}. The output of the general node can then be calculated as
 \begin{equation}
 	y = g(z) = g\left(\sum_i (w_i x_i) + b\right),
 \end{equation}
 where $b$ is the so-called bias of the node which in turn leads to a shift of the weighted sum, irrespective of the inputs. The linear combination $z = \sum_i (w_i x_i) + b$, can also be rewritten as a vector-vector multiplication of vectors $\vec a_{j}^{(k)} = (b, \vec{w})$ and $\vec X = (1, \vec x)$. This leads to a simple expression for the output of a node of $y = g(\vec a \cdot \vec X)$. All parameters of a single node in layer $k$ and position $j$ in the layer are contained within $\vec a_{j}^{(k)}$ \cite{elstnerLectureMachineLearning2025,bishopPatternRecognitionMachine2006}.
 
-We use the term layer, as the output of each layer will be used as the input of the following layer, thus presenting a clear hierarchy. This hierarchy is also useful for calculating the gradient of the final output of the neural network, with respect to all the weights in the layers, as we can use chained differentiation. Calculating the gradient with respect to all the weights is necessary as the optimization techniques presented in \cref{subsec:optimization} rely on the knowledge of the gradients to find the optimal set of weights. The gradient of a single node given the input $\vec{x}$ is given by
+We use the term layer, as the output of each layer will be used as the input of the following layer, thus presenting a clear hierarchy. This hierarchy is also useful for calculating the gradient of the final output of the neural network, with respect to all the weights in the layers, as we can use chained differentiation. Calculating the gradient with respect to all the weights is necessary as the optimization techniques presented in \cref{subsec:optimization} rely on the knowledge of the gradients to find the optimal set of weights. The gradient of a single node with an input $\vec{x}$ is given by
 \begin{equation}
 	\frac{\partial y}{\partial \vec a} = \vec{X} g'(\vec a \vec X),
 \end{equation}
@@ -148,7 +148,7 @@ For simplicity this paper will therefore focus on a set of three activation func
 		0 & z < 0 \\ z & z \geq 0
 	\end{cases}.
 \end{equation}
-While this function provides all necessary properties of an activation function, we will provide an explanation, why it is not well suited for the optimization procedure of FFNN. The issue with the ReLU is that for $z < 0$ the gradient of the function vanishes. This poses a problem, as mentioned above, the optimization algorithm relies on the gradient of the output to adjust and optimize the weights of the neural network. A vanishing gradient disables a possible optimization. This problem is further amplified by the fact, that the gradient of weights in the first layers of the network depends on the product of the gradients with the previous layers, thus a vanishing gradient in one of the last layers has adverse effects on the optimization of weights in the previous layers\cite{kieslerLectureModernMethods2025}.
+While this function provides all necessary properties of an activation function, we will provide an explanation, why it is not well suited for the optimization procedure of FFNN. The issue with the ReLU is that for $z < 0$ the gradient of the function vanishes. This poses a problem, as mentioned above, the optimization algorithm relies on the gradient of the output to adjust and optimize the weights of the neural network. A vanishing gradient disables a possible optimization. This problem is further amplified by the fact, that the gradient of weights in the first layers of the network depends on the product of the gradients with the previous layers, thus a vanishing gradient in one of the last layers has adverse effects on the optimization of weights in the previous layers \cite{kieslerLectureModernMethods2025}.
 
 \paragraph{Leaky Rectified Linear Unit}
 As the ReLU is otherwise quite optimal due to its simplicity, we will mailny use a slightly modified version of the ReLU for the majority of nodes in the studied neural networks, namely the Leaky ReLU (LReLU). The leaky ReLU modifies the standard ReLU by introducing a very small but non-vanishing gradient for $z < 0$. It is defined as
@@ -157,7 +157,7 @@ As the ReLU is otherwise quite optimal due to its simplicity, we will mailny use
 		\epsilon z & z < 0 \\ z & z \geq 0
 	\end{cases},
 \end{equation}
-with $\epsilon$ being the leak coefficient, a small number, set to $\epsilon = \num{e-5}$ for all studies. Due to the step in the derivative at $z = 0$ our function is still nonlinear, but providing a positive derivative over the whole domain\cite{elstnerLectureMachineLearning2025}.
+with $\epsilon$ being the leak coefficient, a small number, set to $\epsilon = \num{e-5}$ for all studies. Due to the step in the derivative at $z = 0$ our function is still nonlinear, but providing a positive derivative over the whole domain \cite{elstnerLectureMachineLearning2025}.
 
 \paragraph{Softmax}
 The last activation to introduce for our studies will be the Softmax function. It shall be defined as \cite{kieslerLectureModernMethods2025}
@@ -168,7 +168,7 @@ Additionally, to providing a source of nonlinearity, the softmax function has th
 
 
 \subsection{Loss Functions}
-The function of the loss function is to quantize the discrepancy between the prediction of a numerical model and the true values\cite{elstnerLectureMachineLearning2025,kieslerLectureModernMethods2025,goodfellowDeepLearning2016}. It is also sometimes called cost function. It must be a differentiable function which is minimal when, the prediction aligns with the true values. The choice of loss function needs to be specific to the task for which the model should be optimized for. For the regressional part of this paper we will use the mean squared error (MSE) as a loss function. The MSE is defined using the prediction $\pred$ and the true values $\true$ as
+The function of the loss function is to quantize the discrepancy between the prediction of a numerical model and the true values \cite{elstnerLectureMachineLearning2025,kieslerLectureModernMethods2025,goodfellowDeepLearning2016}. It is also sometimes called cost function. It must be a differentiable function which is minimal when, the prediction aligns with the true values. The choice of loss function needs to be specific to the task for which the model should be optimized for. For the regressional part of this paper we will use the mean squared error (MSE) as a loss function. The MSE is defined using the prediction $\pred$ and the true values $\true$ as
 \begin{equation}
 	\mathrm{MSE} = \frac{\sum_i (\pred_i - \true_i)^2}{N}.
 \end{equation}
@@ -176,7 +176,7 @@ Its derivative is therefore defined as
 \begin{equation}
 	\frac{\partial \mathrm{MSE}}{\partial \pred_i} = \frac{2 (\pred_i - \true_i)}{N}.
 \end{equation}
-The advantage of the MSE as a loss function is, that the loss value can be directly interpreted in the context of the metric on which we optimize\cite{elstnerLectureMachineLearning2025}. Furthermore, it gives the advantage, that compared to the mean absolute error, which uses the absolute deviation between the target and the prediction, that it is continuous differentiable and that large deviations are penalized more than small deviations. Especially in the context of small measurement uncertainties this is important.
+The advantage of the MSE as a loss function is, that the loss value can be directly interpreted in the context of the metric on which we optimize \cite{elstnerLectureMachineLearning2025}. Furthermore, it gives the advantage, that compared to the mean absolute error, which uses the absolute deviation between the target and the prediction, that it is continuous differentiable and that large deviations are penalized more than small deviations. Especially in the context of small measurement uncertainties this is important.
 
 The second loss function used is the cross entropy loss. It is used for the case of classification tasks and is defined as \cite{elstnerLectureMachineLearning2025}
 \begin{equation}
@@ -230,7 +230,7 @@ The most important constant to tune in the optimization procedure is the learnin
 
 \subsection{Data Splitting Techniques}
 \subsubsection{Train-Test Splitting}
-As we mentioned before, there is the possibility for our model to overadjust for the datapoints present during the optimization process. To still get an accurate measure for the models performance we will only use a subset of the dataset for the process of training. A random subset of \qty{20}{\percent} is reserved for testing the model only. This means, that all performance metrics are evaluated in the same way as new unseen data would perform \cite{kieslerLectureModernMethods2025}. Due to the random nature of the subset, the performance between runs may differ stochastically. This may be counteracted by redoing the tests with different choices of training and testing datasets and averaging the metrics.
+As we mentioned before, there is the possibility for our model to overadjust for the datapoints present during the optimization process. To still get an accurate measure for the models performance we will only use a subset of the dataset for the process of training. A random subset of \qty{20}{\percent} is reserved for only testing the model. This means, that all performance metrics are evaluated in the same way as new unseen data would perform \cite{kieslerLectureModernMethods2025}. Due to the random nature of the subset, the performance between runs may differ stochastically. This may be counteracted by redoing the tests with different choices of training and testing datasets and averaging the metrics.
 
 \subsubsection{Out-of-Fold prediction} \label{subsec:outoffold}
 \begin{figure}
@@ -308,11 +308,11 @@ As a further validation of the functionality, we use the Runge function
 	f(x) = \frac{1}{1 + 25 x^2},
 \end{equation}
 with added Gaussian noise of standard deviation \num{1.0} to create a regression dataset. On this dataset of size $N = \num{100000}$ and \qty{80}{\percent}-\qty{20}{\percent} train-test-splitting we achieve a MSE of \numrange{0.9136}{0.9178}, using between one and two hidden layers with either 50 or 100 nodes per hidden layer. This exceeds the performance of our previous work on least squares regression using different regularization and optimization techniques \cite{bognerRegularizationOptimizationAll2025}, where a MSE of \num{0.92} was achieved using the same dataset \footnote{The $x$ values were expanded into multiple of features containing polynomial multiples $x^n$ of themselves. For optimal performance $n \in \{0, 1, \dots, \geq 10\}$ were required.} and train-test-splitting. 
-In the case of no added noise, the difference is even more stark. In the case of 2 hidden layers with Leaky ReLU activation and 100 nodes per hidden layer on $N= \num{1000}$ points, we achieve $\mathrm{MSE} = \num{1.037e-5}$. OLS methods achieve a performance of \num{0.01984} in this case. \Cref{fig:runge_comparison} shows the predictions by both methods in form of a plot. This motivates further the use of neural networks for regression tasks, as possible non-linearities in the data can be modeled more easily. Least squares methods on the other hand provide only linear combinations of the input features.
+In the case of no added noise, the difference is even more stark. In the case of 2 hidden layers with Leaky ReLU activation and 100 nodes per hidden layer on $N= \num{1000}$ points, we achieve $\mathrm{MSE} = \num{1.037e-5}$. OLS methods achieve a performance of \num{0.01984} in this case. \Cref{fig:runge_comparison} shows the predictions by both methods in form of a plot. Furthermore, this motivates the use of neural networks for regression tasks, as possible non-linearities in the data can be modeled more easily. Least squares methods on the other hand provide only linear combinations of the input features.
 
 
 \subsection{Use of AI tools}
-During the writing of this paper artificial intelligence in the form of large-language models (LLMs) has been used. Two different LLMs have been used for a faster writing of the code used. Mainly for short rewrites of code structure and brain-storming the model GPT-5 by OpenAI and the model DeepSeek by Hangzhou DeepSeek Artificial Intelligence have been used. At no point was artificial intelligence used for writing the report. All output by LLMs has be checked by the author.
+During the creation of this paper artificial intelligence in form of large-language models (LLMs) has been used. Two different LLMs have been used for faster writing of the code used. Mainly for short rewrites of code structure and brain-storming the model GPT-5 by OpenAI and the model DeepSeek by Hangzhou DeepSeek Artificial Intelligence have been used. At no point, was artificial intelligence used for writing the report. All output by LLMs has be checked by the author.
 
 \section{Results and Discussion}\label{section:results}
 \subsection{Introduction to the Dataset}
@@ -323,7 +323,7 @@ During the writing of this paper artificial intelligence in the form of large-la
 	\label{fig:violinplot}
 \end{figure}
 
-For the entire study we use the dataset presented by Wolberg \cite{williamwolbergBreastCancerWisconsin1993}. It includes features recorded from nuclei in breast tissue and their geometric properties. Each physical feature is present three times, where the means and standard deviations of the measured physical features are used as separate features. Furthermore, the mean of the three largest measurements of each physical feature is present in the dataset. The physical features include: the radius, the standard deviation of the images gray-scale values (called texture), the perimeter, the area and the local variation in the radius length, called smoothness. Additionally, the concavity, the number of concave points, the compactness, the symmetry and the fractal dimension are present. For normalization, all features are rescaled to be centered around 0 and have a standard deviation of 1. The normalizaiton is achieved using the \texttt{StandardScaler} class of the \texttt{scikit-learn} library. A quantitative overview on the average of each of the physical features is presented in \cref{fig:violinplot}. The violinplot shows the feature distribution for both classes of benign and malignant cell nuclei.
+For the entire study we use the dataset presented by Wolberg \cite{williamwolbergBreastCancerWisconsin1993}. It includes features recorded from nuclei in breast tissue and their geometric properties. Each physical feature is present three times, where the means and standard deviations of the measured physical features are used as separate features. Furthermore, the mean of the three largest measurements of each physical feature is present in the dataset. The physical features include: the radius, the standard deviation of the images gray-scale values (called texture), the perimeter, the area and the local variation in the radius length (called smoothness). Additionally, the concavity, the number of concave points, the compactness, the symmetry and the fractal dimension are present. For normalization, all features are rescaled to be centered around 0 and have a standard deviation of 1. The normalizaiton is achieved using the \texttt{StandardScaler} class of the \texttt{scikit-learn} library. A quantitative overview on the average of each of the physical features is presented in \cref{fig:violinplot}. The violinplot shows the feature distribution for both classes of benign and malignant cell nuclei.
 
 The separability of the two classes is easily visible given all features in the class. On the other hand, there is no single feature which allows for very good separation. Thus reproducing separability using only a limited set of features is the challenge of this study.
 
@@ -340,7 +340,7 @@ For all studies we use train-test-splitting with \qty{20}{\percent} of the data
 	\label{fig:regression_hyperparameter}
 \end{figure}
 As mentioned in \cref{subsec:activation}, the Leaky ReLU combines the disadvantages of the ReLU while conserving non-vanishing gradients on the entire domain. Thus, it is not surprising, that they perform similar in our studies with the LReLU providing a more stable training process. Due to this finding we will use the LReLU for all the hidden layers.
-\Cref{fig:regression_hyperparameter} shows the resulting MSE for different numbers of hidden layers and numbers of nodes in the layers. We can observe a general trend of decrease in MSE with a decrease in the number of hidden layers and an increase in the number of neurons per hidden layer. This general trend is to be expected as with an increase in the number of hidden layers the gradients in the last layers can converge towards zero with deep networks due to the multiplicic nature of the gradients. Additionally, can an increase number of nodes per layer better capture even complicated relationships in the data. Due to the adverse effects of an increase in node number, we chose a conservative set of parameters with 32 neurons per hidden layer and two hidden layers.
+\Cref{fig:regression_hyperparameter} shows the resulting MSE for different numbers of hidden layers and numbers of nodes in the layers. We can observe a general trend of decrease in MSE with a decrease in the number of hidden layers and an increase in the number of neurons per hidden layer. This general trend is to be expected as with an increase in the number of hidden layers the gradients in the last layers can converge towards zero with deep networks due to the multiplicic nature of the gradients. Additionally, an increase number of nodes per layer can better capture even complicated relationships in the data. Due to the adverse effects of an increase in node number, we chose a conservative set of parameters with 32 neurons per hidden layer and two hidden layers.
 
 In a second sub-study we observe the influence of the regularization constant with and L2-regularization on all layers. Further studies on variable regularization per layer were not carried out. The studies show a steady MSE for a range of \numrange{0}{e-3} with a sharp increase by a factor of \num{10} with the step to $\lambda = \num{e-2}$. Thus, we choose to use $\lambda = \num{e-3}$ for the regressional model. This provides a stable insurance against overfitting of the training data while at the same time not negatively effecting the models' generalization ability.
 
@@ -377,7 +377,7 @@ The excellent performance of the linear regression suggests that the additional
 	\caption{Confusion Matrix of the classifier for the regression output. A target value of 0 indicates malignant nuclei, while 1 is indicative of benign samples.}
 	\label{fig:confusion_matrix}
 \end{figure}
-Similar to the structure of the regression model, we use a FFNN using a variable number of nodes per hidden layer in configurations with different numbers of hidden layers to create a classifier. To use the model as a classifier, the loss function of the model is changed to the cross-entropy loss introduced in \cref{section:methods}. Additionally, we change the activation of the output layer to the softmax function. The leaky rectified linear unit is again used for the purpose in all other neurons, following the results from the previous study. The parameters are optimized using loss minimization using the Adam optimization algorithm.
+Similar to the structure of the regression model, we use a FFNN using a variable number of nodes per hidden layer in configurations with different numbers of hidden layers to create a classifier. To use the model as a classifier, the loss function of the model is changed to the cross-entropy loss introduced in \cref{section:methods}. Additionally, we change the activation of the output layer to the softmax function. The leaky rectified linear unit is again used for the purpose in all other neurons, following the results from the previous study. The parameters are optimized through loss minimization using the Adam optimization algorithm.
 
 As above, a parameter scan is used to find the best parameter combination for accurate prediction, whether the nuclei are part of benign or malignant tissue. The results of the parameter scan are displayed in \cref{fig:classification_hyperparameters}. For the number of hidden layers and neurons, there is a clear maximum in the area-under-curve (AUC) score for the case of 64 neurons in either a single or two hidden layers.  Using the full dataset in the form without regression AUC scores of \num{0.9962} have been shown using logistic regression \cite{mostafaBreastCancerPrediction}. To account for stochastic variations in the training success the AUC is averaged over a total of \num{50} runs for each entry in \cref{fig:classification_hyperparameters}. The regularization constant is chosen to be $\lambda = \num{e-6}$ according to the figure. During \num{50} runs this combination of hyperparameters is able to achieve an average accuracy of \qty{92.70}{\percent} and an AUC score of \num{0.9781}. While this performance is not as good as the performance using logistic regression on the full dataset \cite{mostafaBreastCancerPrediction}, it is to be expected, that the performance decreases when using only a limited set of features to create the regression output and following that the classification.
 
@@ -404,7 +404,7 @@ Additionally, we compare the confusion matrix in \cref{fig:confusion_logistic} a
 \section{Conclusion}\label{section:conclusion}
 It is evident, that a classification performance close to the one using the full dataset can be achieved using only a limited set of physical features. Using a FFNN based regression model followed by a logistic regression classifier, an accuracy of \qty{94.74}{\percent} and an AUC score of \num{0.99105} is achieved. This performance is comparable to the performance of logistic regression on the full dataset \cite{mostafaBreastCancerPrediction}. Using only six features derived from two physical measurements, the radius and area of cell nuclei, this performance is possible.
 
-It is therefore possible to reduce the data acquisition for future diagnostic purposes to a small set of physical features, while at the same time conserving a high classification accuracy. This is especially important in the context of fast and cheap diagnostic procedures.
+It is therefore possible to reduce the data acquisition future diagnostic purposes to a small set of physical features, while at the same time conserving a high classification accuracy. This is especially important in the context of fast and cheap diagnostic procedures.
 
 Additionally, we have shown, that for the regression task on the limited feature set, the performance of ordinary least squares regression is comparable to the one of a carefully optimized FFNN. Thus, for similar tasks in the future, it may be possible to use ordinary least squares regression due to its simplicity and analytical solution.