lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update with log regression

Browse Source
This commit is contained in:
mhjensen
2018-09-14 05:36:58 +02:00
parent 73474229be
commit df4a9d7f27
137 changed files with 79934 additions and 15967 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Regression/html/Regression.html
+407 -113
View File
@@ -219,32 +219,46 @@ div { text-align: justify; text-justify: inter-word; }
'___sec77'),
('Resampling methods: Jackknife', 2, None, '___sec78'),
('Resampling methods: Jackknife estimator', 2, None, '___sec79'),
('Resampling methods: Jackknife sample code',
2,
None,
'___sec80'),
('Resampling methods: Bootstrap', 2, None, '___sec81'),
('Resampling methods: Bootstrap background', 2, None, '___sec82'),
('Resampling methods: Bootstrap', 2, None, '___sec80'),
('Resampling methods: Bootstrap background', 2, None, '___sec81'),
('Resampling methods: More Bootstrap background',
2,
None,
'___sec83'),
('Resampling methods: Bootstrap approach', 2, None, '___sec84'),
('Resampling methods: Bootstrap algorithm', 2, None, '___sec85'),
('Resampling methods: Bootstrap steps', 2, None, '___sec86'),
('Resampling methods: Blocking', 2, None, '___sec87'),
('Blocking Transformations', 2, None, '___sec88'),
('Blocking Transformations', 2, None, '___sec89'),
('Blocking Transformations, getting there', 2, None, '___sec90'),
'___sec82'),
('Resampling methods: Bootstrap approach', 2, None, '___sec83'),
('Resampling methods: Bootstrap steps', 2, None, '___sec84'),
('Resampling methods: Blocking', 2, None, '___sec85'),
('Blocking Transformations', 2, None, '___sec86'),
('Blocking Transformations', 2, None, '___sec87'),
('Blocking Transformations, getting there', 2, None, '___sec88'),
('Blocking Transformations, final expressions',
2,
None,
'___sec91'),
'___sec89'),
('"Code examples for Blocking, Jackknife and '
'bootstrap":"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Programs/ResamplingAnalysisScripts"',
2,
None,
'___sec92')]}
'___sec90'),
('The bias-variance tradeoff', 2, None, '___sec91'),
('Training and testing data', 2, None, '___sec92'),
('Procedure to find a predictor', 2, None, '___sec93'),
('What we want', 2, None, '___sec94'),
('The expected generalization error', 2, None, '___sec95'),
('Elaborating a little bit more', 2, None, '___sec96'),
('The bias', 2, None, '___sec97'),
('The variance', 2, None, '___sec98'),
('Summing up', 2, None, '___sec99'),
('Logistic Regression', 2, None, '___sec100'),
('Basics', 2, None, '___sec101'),
('Linear classifier', 2, None, '___sec102'),
('Some selected properties', 2, None, '___sec103'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec104'),
('Maximum likelihood', 2, None, '___sec105'),
('Minimizing the cross entropy', 2, None, '___sec106')]}
end of tocinfo -->
<body>
@@ -286,7 +300,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 13, 2018</h4></center> <!-- date -->
<center><h4>Sep 14, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2715,18 +2729,18 @@ need for bootstrapping.
<h2 id="___sec78">Resampling methods: Jackknife </h2>
<p>
The Jackknife works by making many replicas of the estimator \( \widehat{\vec{\theta}} \).
The jackknife is a resampling method, we explained that this happens by scrambling the data in some way. When using the jackknife, this is done by systematically leaving out one observation from the vector of observed values \( \vec{X} = (X_1,X_2,\cdots,X_n) \).
Let \( \vec{X}_i \) denote the vector
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
The jackknife is a resampling method, we explained that this happens by scrambling the data in some way. When using the jackknife, this is done by systematically leaving out one observation from the vector of observed values \( \hat{x} = (x_1,x_2,\cdots,X_n) \).
Let \( \hat{x}_i \) denote the vector
$$
\vec{X}_i = (X_1,X_2,\cdots,X_{i-1},X_{i+1},\cdots,X_n),
\hat{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n),
$$
<p>
which equals the vector \( \vec{X} \) with the exception that observation
which equals the vector \( \hat{x} \) with the exception that observation
number \( i \) is left out. Using this notation, define
\( \widehat{\vec{\theta}}_i \) to be the estimator
\( \widehat{\vec{\theta}} \) computed using \( \vec{X}_i \).
\( \widehat{\theta}_i \) to be the estimator
\( \widehat{\theta} \) computed using \( \vec{X}_i \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2735,8 +2749,8 @@ number \( i \) is left out. Using this notation, define
<p>
To get an estimate for the bias and
standard error of \( \widehat{\vec{\theta}} \), use the following
estimators for each component of \( \widehat{\vec{\theta}} \)
standard error of \( \widehat{\theta} \), use the following
estimators for each component of \( \widehat{\theta} \)
$$
\widehat{\mathrm{Bias}}(\widehat \theta,\theta) = (n-1)\left( - \widehat{\theta} + \frac{1}{n}\sum_{i=1}^{n} \widehat \theta_i \right) \qquad \text{and} \qquad \widehat{\sigma}^2_{\widehat{\theta} } = \frac{n-1}{n}\sum_{i=1}^{n}( \widehat{\theta}_i - \frac{1}{n}\sum_{j=1}^{n}\widehat \theta_j )^2.
@@ -2745,43 +2759,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec80">Resampling methods: Jackknife sample code </h2>
<p>
Sample code for the Jackknife method
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">jack</span>(data, stat):
n <span style="color: #666666">=</span> <span style="color: #008000">len</span>(data); t <span style="color: #666666">=</span> zeros(n); inds <span style="color: #666666">=</span> arange(n); t0 <span style="color: #666666">=</span> time()
<span style="color: #408080; font-style: italic"># &#39;jackknifing&#39; by leaving out an observation for each i</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(n):
t[i] <span style="color: #666666">=</span> stat(delete(data,i) )
<span style="color: #008000; font-weight: bold">return</span> t
<span style="color: #408080; font-style: italic"># define a function which returns your chosen estimator theta-hat</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">stat</span>(data):
theta<span style="color: #666666">-</span>hat <span style="color: #666666">=</span> mean(data)
<span style="color: #008000; font-weight: bold">return</span> theta<span style="color: #666666">-</span>hat
<span style="color: #408080; font-style: italic"># Return the Jackknife sample</span>
t <span style="color: #666666">=</span> jack(X, stat)
</pre></div>
<p>
Consider first the function <b>jack()</b>. This function repeatedly
estimates the function called <b>statistic()</b> under the resampled
data by systematically leaving out one observation from the data. The
function <b>stat()</b> is passed as an argument to
<b>jack()</b>. The array <b>t</b> is eventually returned, which
contains all the estimates \( \widehat{\vec{\theta}} \), and can be
plotted or analysed in other ways, such as by calling <b>std(t)</b>
from <b>numpy</b> to estimate the standard error of
\( \widehat{\vec{\theta}} \). The function <b>std(t)</b> is just the
estimator \( \widehat{\sigma}^2 \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec81">Resampling methods: Bootstrap </h2>
<h2 id="___sec80">Resampling methods: Bootstrap </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2802,26 +2780,26 @@ advantages:
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec82">Resampling methods: Bootstrap background </h2>
<h2 id="___sec81">Resampling methods: Bootstrap background </h2>
<p>
Since \( \widehat{\vec{\theta}} = \widehat{\vec{\theta}}(\vec{X}) \) is a function of random variables,
\( \widehat{\vec{\theta}} \) itself must be a random variable. Thus it has
Since \( \widehat{\theta} = \widehat{\theta}(\hat{X}) \) is a function of random variables,
\( \widehat{\theta} \) itself must be a random variable. Thus it has
a pdf, call this function \( p(\vec{t}) \). The aim of the bootstrap is to
estimate \( p(\vec{t}) \) by the relative frequency of
\( \widehat{\vec{\theta}} \). You can think of this as using a histogram
in the place of \( p(\vec{t}) \). If the relative frequency closely
estimate \( p(\hat{t}) \) by the relative frequency of
\( \widehat{\theta} \). You can think of this as using a histogram
in the place of \( p(\hat{t}) \). If the relative frequency closely
resembles \( p(\vec{t}) \), then using numerics, it is straight forward to
estimate all the interesting parameters of \( p(\vec{t}) \) using point
estimate all the interesting parameters of \( p(\hat{t}) \) using point
estimators.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec83">Resampling methods: More Bootstrap background </h2>
<h2 id="___sec82">Resampling methods: More Bootstrap background </h2>
<p>
In the case that \( \widehat{\vec{\theta}} \) has
In the case that \( \widehat{\theta} \) has
more than one component, and the components are independent, use the
same estimator on each component separately. If the probability
density function of \( X_i \), \( p(x) \), had been known, then it would have
@@ -2829,18 +2807,18 @@ been straight forward to do this by:
<ol>
<li> Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).</li>
<li> Then using these numbers, we could compute a replica of \( \widehat{\vec{\theta}} \) called \( \widehat{\vec{\theta}}^* \).</li>
<li> Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).</li>
</ol>
By repeated use of (1) and (2), many
estimates of \( \widehat{\vec{\theta}} \) could have been obtained. The
idea is to use the relative frequency of \( \widehat{\vec{\theta}}^* \)
(think of a histogram) as an estimate of \( p(\vec{t}) \).
estimates of \( \widehat{\theta} \) could have been obtained. The
idea is to use the relative frequency of \( \widehat{\theta}^* \)
(think of a histogram) as an estimate of \( p(\hat{t}) \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec84">Resampling methods: Bootstrap approach </h2>
<h2 id="___sec83">Resampling methods: Bootstrap approach </h2>
<p>
But
@@ -2856,44 +2834,20 @@ result in some asymptotic sense? The answer is yes.
Instead of generating the histogram for the relative
frequency of the observation \( X_i \), just draw the values
\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector
\( \vec{X} \).
\( \hat{X} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec85">Resampling methods: Bootstrap algorithm </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">boot</span>(data, statistic, R):
t <span style="color: #666666">=</span> zeros(R); n <span style="color: #666666">=</span> <span style="color: #008000">len</span>(data); inds <span style="color: #666666">=</span> arange(n); t0 <span style="color: #666666">=</span> time()
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(R):
t[i] <span style="color: #666666">=</span> statistic(data[randint(<span style="color: #666666">0</span>,n,n)])
<span style="color: #008000; font-weight: bold">return</span> t
<span style="color: #408080; font-style: italic"># define a function which returns your chosen estimator theta-hat</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">stat</span>(data):
theta<span style="color: #666666">-</span>hat <span style="color: #666666">=</span> mean(data)
<span style="color: #008000; font-weight: bold">return</span> theta<span style="color: #666666">-</span>hat
t <span style="color: #666666">=</span> boot(X, stat, <span style="color: #666666">2**9</span>)
</pre></div>
<p>
Consider first the function <b>boot()</b>. In the <b>for</b> loop, this function repeatedly estimates the function called <b>statistic()</b> under the resampled data in <b>data(randint(0,n,n))</b>. The function <b>statistic()</b> is passed as an argument to <b>boot()</b>. The array <b>t</b> is eventually returned, which contains all the estimates \( \widehat{\vec{\theta}} \), and can be plotted or analysed in other ways, such as by calling <b>std(t)</b> from <b>numpy</b> to estimate the standard error of \( \widehat{\vec{\theta}} \). The function <b>std(t)</b> is just the estimator \( \widehat{\sigma}^2 \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec86">Resampling methods: Bootstrap steps </h2>
<h2 id="___sec84">Resampling methods: Bootstrap steps </h2>
<p>
The independent bootstrap works like this:
<ol>
<li> Draw with replacement \( n \) numbers for the observed variables \( \vec{x} = (x_1,x_2,\cdots,x_n) \).</li>
<li> Define a vector \( \vec{x}^* \) containing the values which were drawn from \( \vec{x} \).</li>
<li> Using the vector \( \vec{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \vec{x}^* \).</li>
<li> Draw with replacement \( n \) numbers for the observed variables \( \hat{x} = (x_1,x_2,\cdots,x_n) \).</li>
<li> Define a vector \( \hat{x}^* \) containing the values which were drawn from \( \hat{x} \).</li>
<li> Using the vector \( \hat{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \hat{x}^* \).</li>
<li> Repeat this process \( k \) times.</li>
</ol>
@@ -2902,7 +2856,7 @@ When you are done, you can draw a histogram of the relative frequency of \( \wid
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec87">Resampling methods: Blocking </h2>
<h2 id="___sec85">Resampling methods: Blocking </h2>
<p>
The blocking method was made popular by <a href="https://aip.scitation.org/doi/10.1063/1.457480" target="_blank">Flyvbjerg and Pedersen (1989)</a>
@@ -2915,7 +2869,7 @@ Assume \( n = 2^d \) for some integer \( d>1 \) and \( X_1,X_2,\cdots, X_n \) is
Moreover, assume that the time series is asymptotically uncorrelated. We switch to vector notation by arranging \( X_1,X_2,\cdots,X_n \) in an \( n \)-tuple. Define:
$$
\begin{align*}
\vec{X} = (X_1,X_2,\cdots,X_n).
\hat{X} = (X_1,X_2,\cdots,X_n).
\end{align*}
$$
@@ -2928,7 +2882,7 @@ moreover, it becomes more accurate the larger \( n \) is.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec88">Blocking Transformations </h2>
<h2 id="___sec86">Blocking Transformations </h2>
We now define
blocking transformations. The idea is to take the mean of subsequent
pair of elements from \( \vec{X} \) and form a new vector
@@ -2965,7 +2919,9 @@ elements of \( \vec{X}_i \) and let \( n_i \) be the number of elements of
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec89">Blocking Transformations </h2>
<h2 id="___sec87">Blocking Transformations </h2>
<p>
Using the
definition of the blocking transformation and the distributive
property of the covariance, it is clear that since \( h =|i-j| \)
@@ -2987,7 +2943,7 @@ The quantity \( \vec{X} \) is asymptotic uncorrelated by assumption, \( \vec{X}_
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec90">Blocking Transformations, getting there </h2>
<h2 id="___sec88">Blocking Transformations, getting there </h2>
We have
$$
\begin{align}
@@ -3009,7 +2965,7 @@ We can show that \( V(\overline{X}_i) = V(\overline{X}_j) \) for all \( 0 \leq i
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec91">Blocking Transformations, final expressions </h2>
<h2 id="___sec89">Blocking Transformations, final expressions </h2>
<p>
We can then wrap up
@@ -3040,7 +2996,7 @@ It means we can apply blocking transformations until
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec92"><a href="https://github.com/CompPhysics/MachineLearning/tree/master/doc/Programs/ResamplingAnalysisScripts" target="_blank">Code examples for Blocking, Jackknife and bootstrap</a> </h2>
<h2 id="___sec90"><a href="https://github.com/CompPhysics/MachineLearning/tree/master/doc/Programs/ResamplingAnalysisScripts" target="_blank">Code examples for Blocking, Jackknife and bootstrap</a> </h2>
<p>
@@ -3270,6 +3226,344 @@ dataAnalysis<span style="color: #666666">.</span>plotAll()
dataAnalysis<span style="color: #666666">.</span>printOutput()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec91">The bias-variance tradeoff </h2>
We begin with an unknown function \( y=f(x) \) and fix a \emph{hypothesis set}
\( \mathcal{H} \) consisting of all functions we are willing to consider,
defined also on the domain of \( f \). This set may be uncountably
infinite (e.g.&nbsp;if there are real-valued parameters to fit).
The
choice of which functions to include in \( \mathcal{H} \) usually depends
on our intuition about the problem of interest. The function \( f(x) \)
produces a set of pairs \( (x_i,y_i) \), \( i=1\dots N \), which serve as the
observable data. Our goal is to select a function from the hypothesis
set \( h\in\mathcal{H} \) which approximates \( f(x) \) as best as possible,
namely, we would like to find \( h\in\mathcal{H} \) such that \( h\approx
f \) in some strict mathematical sense which we specify below. If this
is possible, we say that we \emph{learned} \( f(x) \). But if the
function \( f(x) \) can, in principle, take any value on
\emph{unobserved} inputs, how is it possible to learn in any
meaningful sense?
<p>
<!-- !split -->
<h2 id="___sec92">Training and testing data </h2>
<p>
We will discuss the bias-variance tradeoff in the context of continuous predictions such as regression. However, many of the intuitions and ideas discussed here also carry over to classification tasks. Consider a dataset \( \mathcal{L} \) consisting of the data \( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=1\ldots N\} \). Let us assume that the true data is generated from a noisy model
$$
y=f(\boldsymbol{x}) + \epsilon
$$
where \( \epsilon \) is normally distributed with mean zero and standard deviation \( \sigma_\epsilon \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec93">Procedure to find a predictor </h2>
<p>
We have a statistical procedure (e.g. least-squares regression) for
forming a predictor \( \hat{g}_{\mathcal{L}}(\boldsymbol{x}) \) that gives the
prediction of our model for a new data point \( \boldsymbol{x} \). This estimator
is chosen by minimizing a cost function which we take to be the
squared error
$$
\mathcal{C}( \boldsymbol{X}, \hat{g}(\boldsymbol{x})) = \sum_i (y_i - \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec94">What we want </h2>
<p>
We are interested in the generalization error on all data drawn from
the true model, not just the error on the particular training dataset
\( \mathcal{L} \) that we have in hand. This is just the expectation of
the cost function over many different data sets
\( \{\mathcal{L}_j\} \). Denote this expectation value by
\( E_{\mathcal{L}} \). In other words, we can view \( \hat{g}_{\mathcal{L}} \)
as a stochastic functional that depends on the dataset \( \mathcal{L} \)
and we can think of \( E_{\mathcal{L}} \) as the expected value of the
functional if we drew an infinite number of datasets \( \{\mathcal{L}_1,
\mathcal{L}_2, \ldots \} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec95">The expected generalization error </h2>
<p>
We would also like to average over different instances of the
&quot;noise&quot; \( \epsilon \) and we denote the expectation value over the
noise by \( E_\epsilon \). Thus, we can decompose the expected
generalization error as
$$
\begin{align}
E_\mathcal{L, \epsilon}[\mathcal{C}( \boldsymbol{X}, \hat{g}(\boldsymbol{x})) ]&= E_\mathcal{L,\epsilon}\left[ \sum_i ({y}_i - \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2 \right] \nonumber \\
&= E_\mathcal{L, \epsilon}\left[ \sum_{i}({y}_i -f(\boldsymbol{x}_i) +f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2\right] \nonumber \\
&= \sum_i E_\epsilon[ ({y}_i -f(\boldsymbol{x}_i))^2 ]+ E_\mathcal{L, \epsilon}[(f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2] + 2E_\epsilon[{y}_i -f(\boldsymbol{x}_i)]E_\mathcal{L}[f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i)] \nonumber \\
&=\sum_i \sigma_\epsilon^2 + E_\mathcal{L}[(f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2],
\label{_auto17}
\end{align}
$$
<p>
where in the last line we used the fact that our noise has zero mean
and variance \( \sigma_\epsilon^2 \) and the sum over \( i \) applies to all
terms.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec96">Elaborating a little bit more </h2>
<p>
It is also helpful to further decompose the second term as
follows:
$$
\begin{align}
E_\mathcal{L}[(f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2] &=E_\mathcal{L}[(f(\mathbf{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)]+ E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)]- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2] \nonumber \\
&=E_\mathcal{L}[(f(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2] + E_\mathcal{L}[( \hat{g}_\mathcal{L}(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2] \nonumber \\
&+2E_\mathcal{L}[(f(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])( \hat{g}_\mathcal{L}(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])] \nonumber \\
&=(f(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2+E_\mathcal{L}[( \hat{g}_\mathcal{L}(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2].
\label{_auto18}
\end{align}
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec97">The bias </h2>
The first term is called the bias
$$
Bias^2= \sum_i (f(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2
$$
and measures the deviation of the expectation value of our estimator (i.e. the asymptotic value of our estimator in the infinite data limit) from the true value.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec98">The variance </h2>
The second term is called the variance
$$
Var=\sum_i E_\mathcal{L}[( \hat{g}_\mathcal{L}(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2],
$$
<p>
and measures how much our estimator fluctuates due to finite-sample effects. Combining these expressions, we see that the expected out-of-sample error of our model can be decomposed as
$$
E_\mathrm{out}=E_\mathcal{L, \epsilon}[\mathcal{C}( \boldsymbol{X}, \hat{g}(\boldsymbol{x})) ] = Bias^2 + Var + Noise.
$$
<p>
The bias-variance tradeoff summarizes the fundamental tension in
machine learning, particularly supervised learning, between the
complexity of a model and the amount of training data needed to train
it. Since data is often limited, in practice it is often useful to
use a less-complex model with higher bias &ndash; a model whose asymptotic
performance is worse than another model &ndash; because it is easier to
train and less sensitive to sampling noise arising from having a
finite-sized training dataset (smaller variance).
<p>
<!-- !split -->
<h2 id="___sec99">Summing up </h2>
<p>
The above equations tell us that in
order to minimize the expected test error, we need to select a
statistical learning method that simultaneously achieves low variance
and low bias. Note that variance is inherently a nonnegative quantity,
and squared bias is also nonnegative. Hence, we see that the expected
test MSE can never lie below \( Var(\epsilon) \), the irreducible error.
<p>
What do we mean by the variance and bias of a statistical learning
method? The variance refers to the amount by which our model would change if we
estimated it using a different training data set. Since the training
data are used to fit the statistical learning method, different
training data sets will result in a different estimate. But ideally the
estimate for our model should not vary too much between training
sets. However, if a method has high variance then small changes in
the training data can result in large changes in the model. In general, more
flexible statistical methods have higher variance.
<p>
<!-- !split -->
<h2 id="___sec100">Logistic Regression </h2>
<p>
So far we have focused on learning from datasets for which there is a
<b>continuous</b> output. In linear regression we have been
concerned with learning the coefficients of a polynomial to predict
the response of a continuous variable \( y_i \) on unseen data based on
its independent variables \( {\bf x}_i \).
<p>
Classification problems,
however, are concerned with outcomes taking the form of discrete
variables (i.e. categories). For example, we may want to detect if
there's a cat or a dog in an image. Or given a specific system,
we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics).
(e.g. ordered/disordered).
<p>
<b>Logistic regression deals with binary, dichotomous outcomes (e.g. True or
False, Success or Failure, etc.). It is worth noting that logistic
regression is also commonly used in modern supervised Deep Learning
models</b>, as we will see later.
<p>
<!-- !split -->
<h2 id="___sec101">Basics </h2>
<p>
We consider the case where the dependent variables \( y_i\in\mathbb{Z} \)
are discrete and only take values from \( m=0,\dots,M-1 \) (i.e. \( M \)
classes).
<p>
The goal is to predict the
output classes from the design matrix \( X\in\mathbb{R}^{n\times p} \)
made of \( n \) samples, each of which bears \( p \) features. Of cours e the
primary goal is to identify the classes to which new unseen samples
belong.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec102">Linear classifier </h2>
<p>
Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset
$$
\begin{equation}
s_i = \boldsymbol{x}_i^T\boldsymbol{w} + b_0 \equiv \mathbf{x}_i^T\mathbf{w},
\label{_auto19}
\end{equation}
$$
where we use the short-hand notation
\( \mathbf{x}_i = (1,\boldsymbol{x}_i) \) and \( \mathbf{w}_i = (b_0,\boldsymbol{w}_i) \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec103">Some selected properties </h2>
<p>
This function takes values on the entire real axis. In the case of logistic regression, however, the labels \( y_i \) are discrete variables. One simple way to get a discrete output is to have sign functions that map the output of a linear regressor to \( \{0,1\} \), \( f(s_i)= \) sign$(s_i) = 1$ if \( s_i\ge 0 \) and 0 if otherwise. Indeed, this is commonly known as the &quot;perceptron" in the machine learning literature. This model is extremely simple, and it is favorable in many cases (e.g. noisy data) to have a ``soft" classifier that outputs the probability of a given category. For example, given \( \mathbf{x}_i \), the classifier outputs the probability of being in category \( m \). One such function is the logistic (or sigmoid) function:
$$
\begin{equation}
f(s) = \frac{1}{1+\mathrm e^{-s}}.
\label{eq:log_fun}
\end{equation}
$$
Note that \( 1-f(s)= f(-s) \), which will be useful shortly.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec104">The cross-entropy as a cost function for logistic regression </h2>
<p>
The perceptron is an example of a ``hard classification&quot;: each datapoint is deterministically assigned to a category (i.e \( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a &quot;soft&quot; classifier that outputs the probability of a given category rather than a single value. For example, given \( \mathbf{x}_i \), the classifier outputs the probability of being in category \( m \).
Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point \( \boldsymbol{x}_i \) belongs to a category \( y_i=\{0,1\} \) is is given by
$$
\begin{eqnarray}
P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& \frac{1}{1+\mathrm{e}^{-\mathbf{x}^T_i\mathbf{w}}},\nonumber\\
P(y_i=0|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& 1 - P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)},
\end{eqnarray}
$$
where \( \boldsymbol{\theta}=\mathbf{w} \) are the weights we wish to learn from the data.
<p>
Notice that in terms of the logistic function, we can write
$$
P(y_i=1) =f(\mathbf{x}_i^T\mathbf{w})=1-P(y_i=0).
$$
<p>
<!-- !split -->
<h2 id="___sec105">Maximum likelihood </h2>
<p>
We now define the cost function for logistic regression using Maximum
Likelihood Estimation (MLE). Recall, that in MLE we choose parameters
to maximize the probability of seeing the observed data. Consider a
dataset \( \mathcal{D}=\{(y_i,\boldsymbol{x}_i)\} \) with binary labels
\( y_i\in\{0,1\} \) where the data points are drawn independently. The
likelihood of the seeing the data under our model is just:
$$
\begin{align}
P(\mathcal{D}|\mathbf{w})& = \prod_{i=1}^n \left[f(\mathbf{x}_i^T\mathbf{w})\right]^{y_i}\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]^{1-y_i}\nonumber \\
\label{_auto20}
\end{align}
$$
from which we can readily compute the log-likelihood:
$$
\begin{equation}
l(\mathbf{w}) = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
\label{_auto21}
\end{equation}
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \theta \)
$$
\hat{\mathbf{w}} = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
$$
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
$$
\begin{eqnarray}
\mathcal{C}(\mathbf{w}) &=& - l(\mathbf{w}) \\
&=& \sum_{i=1}^n -y_i\log f(\mathbf{x}_i^T\mathbf{w}) - (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].\nonumber
\end{eqnarray}
$$
This equation is known in statistics as the \emph{cross entropy}. Finally, we note that just as in linear regression,
in practice we usually supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec106">Minimizing the cross entropy </h2>
<p>
The cross entropy is a convex function of the weights \( \mathbf{w} \) and,
therefore, any local minimizer is a global minimizer. Minimizing this
cost function leads to the following equation
$$
\begin{equation}
\boldsymbol{0}=\boldsymbol{\nabla} \mathcal{C}(\mathbf{w}) = \sum_{i=1}^n\left[f(\mathbf{x}_i^T\mathbf{w})-y_i\right]\mathbf{x}_i,
\label{_auto22}
\end{equation}
$$
<p>
where we made use of the logistic function identity \( \partial_z f(z) =
f(z)[1-f(z)] \). This equation defines a transcendental equation for
\( \mathbf{w} \), the solution of which, unlike linear regression, cannot
be written in a closed form.
Here we need gradient descent methods!
<!-- ------------------- end of main content --------------- -->