Logistic regression is a linear model for classification. Recalling the cost function for ordinary least squares with both L2 (ridge) and L1 (LASSO) penalties we will see that the logistic cost function is very similar. In OLS we wish to predict a continuous variable \( \hat{y} \) using $$ \begin{align} \hat{y} = X\omega, \tag{4} \end{align} $$
where \( X \in \mathbb{R}^{n \times p} \) is the input data and \( \omega^{p \times d} \) are the weights of the regression. In a classification setting (binary classification in our situation) we are interested in a positive or negative answer. We can thus define either answer to be above or below some threshold. But, in order to limit the size of the answer and also to get a probability interpretation on how sure we are for either answer we can compute the sigmoid function of OLS. That is, $$ \begin{align} f(X\omega) = \frac{1}{1 + \exp(-X\omega)}. \tag{5} \end{align} $$ We are thus interested in minizming the following cost function $$ \begin{align} C(X, \omega) = \sum_{i = 1}^n \left\{ - y_i\log\left( f(x_i^T\omega) \right) - (1 - y_i)\log\left[1 - f(x_i^T\omega)\right] \right\}, \tag{6} \end{align} $$
where we will restrict ourselves to a value for \( f(z) \) as the sigmoid described above. We can also tack on a L2 (Ridge) or L1 (LASSO) penalization to this cost function in the same manner we did for linear regression.