Minor changes

report/main.tex

@@ -184,6 +184,17 @@ The second loss function used is the cross entropy loss. It is used for the case
 \end{equation}
 The cross entropy loss is inspired by the definition of the physical entropy of stochastic systems and is useful, as it is minimal when the probabilities $\pred$ and $\true$ align. Combined with the softmax activation function from \cref{subsec:activation}, the derivative simplifies to a proportional relation with respect to $\pred - \true$.
 
+\subsubsection{Other Evaluation Metrics}
+Additional to the metrics on which we optimize the model, there are other scores than can be important in the context of evaluating the quality of a models performance. The relative amount of true predictions of a classification model can be judged using the accuracy. The accuracy is defined as
+\begin{equation}
+  \mathrm{Accuracy} = \frac{\sum_{c \in C} N(\true = c, \pred = c)}{N_\mathrm{total}},
+\end{equation}
+where $C$ contains all the possible classes of the dataset and $N$ is the number of predictions with certain conditions. A second metric for evaluating the performance of a classifier is the area-under-curve (AUC) score of the receiver-operating-characteristic (ROC). It is defined as
+\begin{equation}
+  \mathrm{AUC} = \int_0^1 \mathcal{T}(\mathcal{F}) \mathrm d \mathcal F,
+\end{equation}
+where $\mathcal T$ is the true-positive-rate and $\mathcal F$ is the false-positive-rate. A score of 1 indicates perfect classification, while a score of \num{0.5} is equivalent to a random classifier, as in that case $\mathcal T(\mathcal F) \equiv \mathcal F$.
+
 
 \subsection{Regularization}
 As the number of parameters in a neural network grows very quickly, special attention needs to be payed to avoid overfitting the training dataset. Overfitting means, that the available degress of freedom of the model will be used to represent all the datapoints in the training dataset perfectly. This will lead to very poor generalization ability of the model. This phenomenon is part of the bias-variance trade-off, which is ubiquitous in machine learning. With increasing numbers of parameters in the model, i.e. due to an increase in layer size and/or increase in the number of layers, the number of degrees of freedom also increases. With a larger number of degrees of freedom the variance of the model increases. If there are too few parameters, the bias of the model will increase. The goal is therefore to find the middle point, where neither the bias nor variance are increased. 
@@ -215,6 +226,8 @@ To decrease the computational resources needed we will use a more advanced versi
 \end{equation}
 with the optimization parameters $\beta_{1/2}$. As shown in our previous article \cite{bognerRegularizationOptimizationAll2025}, Adam tends to converge faster towards an optimal solution. It is also thus widely used in machine learning optimizations and will be used for all training procedures throughout the studies.
 
+The most important constant to tune in the optimization procedure is the learning rate $\alpha$. It controls the step size of each update of the weights. If it is chosen too large, the optimization may overshoot the optimal solution and diverge. If it is chosen too small, the optimization will take unnecessarily long to converge. A good choice of learning rate is therefore crucial for an efficient optimization procedure. Chosing the learning rate in this study is done by judging the convergence behavior at different learning rates graphically. We then choose a rate which provides fast convergence without divergence and an additional margin of safety.
+
 \subsection{Data Splitting Techniques}
 \subsubsection{Train-Test Splitting}
 As we mentioned before, there is the possibility for our model to overadjust for the datapoints present during the optimization process. To still get an accurate measure for the models performance we will only use a subset of the dataset for the process of training. A random subset of \qty{20}{\percent} is reserved for testing the model only. This means, that all performance metrics are evaluated in the same way as new unseen data would perform. Due to the random nature of the subset, the performance between runs may differ stochastically. This may be counteracted by redoing the tests with different choices of training and testing datasets and averaging the metrics.
@@ -303,6 +316,8 @@ The separability of the two classes is easily visible given all features in the
 \subsection{Regression on Limited Feature Set}
 We use a FFNN with six input nodes and hidden layers of equal node count. For the output we use a layer using linear activation $g(z) = z$, i.e. no activation function or in other words a linear regression. To measure the performance of the model, the MSE of the models outputs is used as a loss metric. The model is fitted using the Adam optimizer algorithm with a learning rate of $\lambda = \num{e-2}$ and decay parameters $\beta_1 = \num{0.9}$ and $\beta_2 = \num{0.999}$. The mean squared error per target variable is evaluated. For a simple model without any hyperparameter optimization the MSE is in the range of \qtyrange{2e-4}{2e1}{\std \squared}, where \unit{\std} signifies the standard deviation of the feature values.
 
+For all studies we use train-test-splitting with \qty{20}{\percent} of the data reserved for testing. This is important as to get an unbiased estimate of the models performance on unseen data.
+
 \subsubsection{Hyperparameter Tuning}
 \begin{figure}
   \centering
@@ -348,18 +363,26 @@ The excellent performance of the linear regression suggests that the additional
 \end{figure}
 Similar to the structure of the regression model, we use a FFNN using a variable number of nodes per hidden layer in configurations with different numbers of hidden layers to create a classifier. To use the model as a classifier, the loss function of the model is changed to the cross-entropy loss introduced in \cref{section:methods}. Additionally, we change the activation of the output layer to the softmax function. The leaky rectified linear unit is again used for the purpose in all other neurons, following the results from the previous study. The parameters are optimized using loss minimization using the Adam optimization algorithm. 
 
-As above, a parameter scan is used to find the best parameter combination for accurate prediction, whether the nuclei are part of benign or malignant tissue. The results of the parameter scan are displayed in \cref{fig:classification_hyperparameters}. For the number of hidden layers and neurons, there is a clear maximum in the area-under-curve (AUC) score for the case of 64 neurons in either a single or two hidden layers. The AUC measures the integral of the receiver-operating-characteristic, which is the graph of true positive rate against the false positive rate. A score of 1 indicates perfect classification, while a score of \num{0.5} is equivalent to a random classifier. Using the full dataset in the form without regression AUC scores of \num{0.9962} have been shown using logisitic regression \cite{mostafaBreastCancerPrediction}. To account for stochastic variations in the training success the AUC is averaged over a total of \num{50} runs for each entry in \cref{fig:classification_hyperparameters}. The regularization constant is chosen to be $\lambda = \num{e-6}$ according to the figure. During \num{50} runs this combination of hyperparameters is able to achieve an average accuracy of \qty{92.70}{\percent} and an AUC score of \num{0.9781}. While this performance is not as good as the performance using logistic regression on the full dataset \cite{mostafaBreastCancerPrediction}, it is to be expected, that the performance decreases when using only a limited set of features to create the regression output and following that the classification.
+As above, a parameter scan is used to find the best parameter combination for accurate prediction, whether the nuclei are part of benign or malignant tissue. The results of the parameter scan are displayed in \cref{fig:classification_hyperparameters}. For the number of hidden layers and neurons, there is a clear maximum in the area-under-curve (AUC) score for the case of 64 neurons in either a single or two hidden layers.  Using the full dataset in the form without regression AUC scores of \num{0.9962} have been shown using logisitic regression \cite{mostafaBreastCancerPrediction}. To account for stochastic variations in the training success the AUC is averaged over a total of \num{50} runs for each entry in \cref{fig:classification_hyperparameters}. The regularization constant is chosen to be $\lambda = \num{e-6}$ according to the figure. During \num{50} runs this combination of hyperparameters is able to achieve an average accuracy of \qty{92.70}{\percent} and an AUC score of \num{0.9781}. While this performance is not as good as the performance using logistic regression on the full dataset \cite{mostafaBreastCancerPrediction}, it is to be expected, that the performance decreases when using only a limited set of features to create the regression output and following that the classification.
 
 The performance is further evaluated using the confusion matrix in \cref{fig:confusion_matrix}. Potential asymmetric biases for false identification, i.e. wether type 1 or 2 errors are more pronounced, can be detected using a confusion matrix. It shows which relative amount of each class (malignant or benign) is identified as which by the classifier. No signifcant asymmetry is present in our FFNN based classifier.
 
 
 \subsubsection{Using Logistic Regression for Classification}
+\begin{figure}
+  \centering
+  \includegraphics[width=\columnwidth]{../notebooks/logistic_classification_confusion_matrix.pdf}
+  \caption{Confusion Matrix of the logistic regression for the FFNN regression output. A target value of 0 indicates malignant nuclei, while 1 is indicative of benign samples.}
+  \label{fig:confusion_logistic}
+\end{figure}
 Thus, we also study the use of logistic regression to create a classification based on the outputs of the regression model. The logistic regression model uses the same framework code as the FFNN for comparability. For creating a logistic regression classification, a singular layer with two neurons and a softmax activation is used. The cross-entropy-loss is used as an optimization metric, which is then minimized using the Adam optimizer. 
 
 After the minimization a test split is evaluated for accuracy and receiver-operating-characteristic leading to an accuracy value of \qty{94.74}{\percent} and an AUC score of \num{0.99105}. The logistic regression thus outperforms the classification models by far. It is especially noteworthy, that nearly the same performance as for full original dataset data can be reached. While it might seem counterintuitive, that the more complex models are not able to match the performance of the arguably very simple logistic regression is due to the fact, that even with regularization, the more complex models have a higher tendency to overfit simple relations between the features and the classification. 
 
 The good performance of the linear regression also helps fur future diagnostic purposes, as we have shown, that with the correct combination of neural networks and logistic regression it is possible to use a very small, easy to extract subset of the physical features to achieve nuclei identification at the highest level.
 
+Additionally, we compare the confusion matrix in \cref{fig:confusion_logistic} against our findings for the FFNN. A bias towards type 2 errors, where malignant tissue is predicted as benign, is evident. Such a misclassification is especially important in the context of diagnostic use.
+
 
 
 \section{Conclusion}\label{section:conclusion}