Morten Hjorth-Jensen
677 lines
38 KiB
HTML
677 lines
38 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/doconce/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
|
|
|
<title></title>
|
|
|
|
|
|
<link href="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
|
|
<script src="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
|
|
<script>hljs.initHighlightingOnLoad();</script>
|
|
|
|
<link href="https://thomasf.github.io/solarized-css/solarized-light.min.css" rel="stylesheet">
|
|
<style type="text/css">
|
|
h1 {color: #b58900;} /* yellow */
|
|
/* h1 {color: #cb4b16;} orange */
|
|
/* h1 {color: #d33682;} magenta, the original choice of thomasf */
|
|
code { padding: 0px; background-color: inherit; }
|
|
pre {
|
|
border: 0pt solid #93a1a1;
|
|
box-shadow: none;
|
|
}
|
|
|
|
div { text-align: justify; text-justify: inter-word; }
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 1,
|
|
'sections': [('Generative Models', 1, None, 'generative-models'),
|
|
('Generative Adversarial Networks',
|
|
2,
|
|
None,
|
|
'generative-adversarial-networks'),
|
|
('Writing Our First Generative Adversarial Network',
|
|
2,
|
|
None,
|
|
'writing-our-first-generative-adversarial-network'),
|
|
('Exploring the Latent Space',
|
|
2,
|
|
None,
|
|
'exploring-the-latent-space'),
|
|
('Interpolating Between MNIST Digits',
|
|
2,
|
|
None,
|
|
'interpolating-between-mnist-digits')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
|
|
<script type="text/x-mathjax-config">
|
|
MathJax.Hub.Config({
|
|
TeX: {
|
|
equationNumbers: { autoNumber: "AMS" },
|
|
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
|
}
|
|
});
|
|
</script>
|
|
<script type="text/javascript" async
|
|
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
|
</script>
|
|
|
|
|
|
|
|
|
|
<!-- ------------------- main content ---------------------- -->
|
|
|
|
<h1 id="generative-models">Generative Models </h1>
|
|
<b>Generative models</b> describe a class of statistical models that are a contrast
|
|
to <b>discriminative models</b>. Informally we say that generative models can
|
|
generate new data instances while discriminative models discriminate between
|
|
different kinds of data instances. A generative model could generate new photos
|
|
of animals that look like 'real' animals while a discriminative model could tell
|
|
a dog from a cat. More formally, given a data set \( x \) and a set of labels /
|
|
targets \( y \). Generative models capture the joint probability \( p(x, y) \), or
|
|
just \( p(x) \) if there are no labels, while discriminative models capture the
|
|
conditional probability \( p(y | x) \). Discriminative models generally try to draw
|
|
boundaries in the data space (often high dimensional), while generative models
|
|
try to model how data is placed throughout the space.
|
|
|
|
<h2 id="generative-adversarial-networks">Generative Adversarial Networks </h2>
|
|
<b>Generative Adversarial Networks</b> are a type of unsupervised machine learning
|
|
algorithm proposed by <a href="https://arxiv.org/pdf/1406.2661.pdf" target="_blank">Goodfellow et. al</a>
|
|
in 2014 (Read the paper first it's only 6 pages). The simplest formulation of
|
|
the model is based on a game theoretic approach, <em>zero sum game</em>, where we pit
|
|
two neural networks against one another. We define two rival networks, one
|
|
generator \( g \), and one discriminator \( d \). The generator directly produces
|
|
samples
|
|
$$
|
|
\begin{equation}
|
|
x = g(z; \theta^{(g)})
|
|
\label{_auto1}
|
|
\end{equation}
|
|
$$
|
|
|
|
The discriminator attempts to distinguish between samples drawn from the
|
|
training data and samples drawn from the generator. In other words, it tries to
|
|
tell the difference between the fake data produced by \( g \) and the actual data
|
|
samples we want to do prediction on. The discriminator outputs a probability
|
|
value given by
|
|
|
|
$$
|
|
\begin{equation}
|
|
d(x; \theta^{(d)})
|
|
\label{_auto2}
|
|
\end{equation}
|
|
$$
|
|
|
|
<p>
|
|
indicating the probability that \( x \) is a real training example rather than a
|
|
fake sample the generator has generated. The simplest way to formulate the
|
|
learning process in a generative adversarial network is a zero-sum game, in
|
|
which a function
|
|
|
|
$$
|
|
\begin{equation}
|
|
v(\theta^{(g)}, \theta^{(d)})
|
|
\label{_auto3}
|
|
\end{equation}
|
|
$$
|
|
|
|
<p>
|
|
determines the reward for the discriminator, while the generator gets the
|
|
conjugate reward
|
|
|
|
$$
|
|
\begin{equation}
|
|
-v(\theta^{(g)}, \theta^{(d)})
|
|
\label{_auto4}
|
|
\end{equation}
|
|
$$
|
|
|
|
<p>
|
|
During learning both of the networks maximize their own reward function, so that
|
|
the generator gets better and better at tricking the discriminator, while the
|
|
discriminator gets better and better at telling the difference between the fake
|
|
and real data. The generator and discriminator alternate on which one trains at
|
|
one time (i.e. for one epoch). In other words, we keep the generator constant
|
|
and train the discriminator, then we keep the discriminator constant to train
|
|
the generator and repeat. It is this back and forth dynamic which lets GANs
|
|
tackle otherwise intractable generative problems. As the generator improves with
|
|
training, the discriminator's performance gets worse because it cannot easily
|
|
tell the difference between real and fake. If the generator ends up succeeding
|
|
perfectly, the the discriminator will do no better than random guessing i.e.
|
|
50\%. This progression in the training poses a problem for the convergence
|
|
criteria for GANs. The discriminator feedback gets less meaningful over time,
|
|
if we continue training after this point then the generator is effectively
|
|
training on junk data which can undo the learning up to that point. Therefore,
|
|
we stop training when the discriminator starts outputting \( 1/2 \) everywhere.
|
|
At convergence we have
|
|
|
|
$$
|
|
\begin{equation}
|
|
g^* = \underset{g}{\mathrm{argmin}}\hspace{2pt}
|
|
\underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)})
|
|
\label{_auto5}
|
|
\end{equation}
|
|
$$
|
|
|
|
The default choice for \( v \) is
|
|
$$
|
|
\begin{equation}
|
|
v(\theta^{(g)}, \theta^{(d)}) = \mathbb{E}_{x\sim p_\mathrm{data}}\log d(x)
|
|
+ \mathbb{E}_{x\sim p_\mathrm{model}}
|
|
\log (1 - d(x))
|
|
\label{_auto6}
|
|
\end{equation}
|
|
$$
|
|
|
|
The main motivation for the design of GANs is that the learning process requires
|
|
neither approximate inference (variational autoencoders for example) nor
|
|
approximation of a partition function. In the case where
|
|
$$
|
|
\begin{equation}
|
|
\underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)})
|
|
\label{_auto7}
|
|
\end{equation}
|
|
$$
|
|
|
|
is convex in $\theta^{(g)} then the procedure is guaranteed to converge and is
|
|
asymptotically consistent
|
|
( <a href="https://arxiv.org/pdf/1804.09139.pdf" target="_blank">Seth Lloyd on QuGANs</a> ). This is in
|
|
general not the case and it is possible to get situations where the training
|
|
process never converges because the generator and discriminator chase one
|
|
another around in the parameter space indefinitely. A much deeper discussion on
|
|
the currently open research problem of GAN convergence is available
|
|
<a href="https://www.deeplearningbook.org/contents/generative_models.html" target="_blank">here</a>. To
|
|
anyone interested in learning more about GANs it is a highly recommended read.
|
|
Direct quote: "In this best-performing formulation, the generator aims to
|
|
increase the log probability that the discriminator makes a mistake, rather than
|
|
aiming to decrease the log probability that the discriminator makes the correct
|
|
prediction." <a href="https://arxiv.org/abs/1701.00160" target="_blank">Another interesting read</a>
|
|
|
|
<h2 id="writing-our-first-generative-adversarial-network">Writing Our First Generative Adversarial Network </h2>
|
|
Let us now move on to actually implementing a GAN in tensorflow. We will study
|
|
the performance of our GAN on the MNIST dataset. This code is based on and
|
|
adapted from the
|
|
<a href="https://www.tensorflow.org/tutorials/generative/dcgan" target="_blank">google tutorial</a>
|
|
|
|
<p>
|
|
First we import our libraries
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">os</span>
|
|
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">time</span>
|
|
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
|
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">tensorflow</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">tf</span>
|
|
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
|
|
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras</span> <span style="color: #8B008B; font-weight: bold">import</span> layers
|
|
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.utils</span> <span style="color: #8B008B; font-weight: bold">import</span> plot_model
|
|
</pre></div>
|
|
<p>
|
|
Next we define our hyperparameters and import our data the usual way
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>BUFFER_SIZE = <span style="color: #B452CD">60000</span>
|
|
BATCH_SIZE = <span style="color: #B452CD">256</span>
|
|
EPOCHS = <span style="color: #B452CD">30</span>
|
|
|
|
data = tf.keras.datasets.mnist.load_data()
|
|
(train_images, train_labels), (test_images, test_labels) = data
|
|
train_images = np.reshape(train_images, (train_images.shape[<span style="color: #B452CD">0</span>],
|
|
<span style="color: #B452CD">28</span>,
|
|
<span style="color: #B452CD">28</span>,
|
|
<span style="color: #B452CD">1</span>)).astype(<span style="color: #CD5555">'float32'</span>)
|
|
|
|
<span style="color: #228B22"># we normalize between -1 and 1</span>
|
|
train_images = (train_images - <span style="color: #B452CD">127.5</span>) / <span style="color: #B452CD">127.5</span>
|
|
training_dataset = tf.data.Dataset.from_tensor_slices(
|
|
train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
|
|
</pre></div>
|
|
<p>
|
|
Let's have a quick look
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>plt.imshow(train_images[<span style="color: #B452CD">0</span>], cmap=<span style="color: #CD5555">'Greys'</span>)
|
|
plt.show()
|
|
</pre></div>
|
|
<p>
|
|
Now we define our two models. This is where the 'magic' happens. There are a
|
|
huge amount of possible formulations for both models. A lot of engineering and
|
|
trial and error can be done here to try to produce better performing models. For
|
|
more advanced GANs this is by far the step where you can 'make or break' a
|
|
model.
|
|
|
|
<p>
|
|
We start with the generator. As stated in the introductory text the generator
|
|
\( g \) upsamples from a random sample to the shape of what we want to predict. In
|
|
our case we are trying to predict MNIST images (\( 28\times 28 \) pixels).
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">generator_model</span>():
|
|
<span style="color: #CD5555">"""</span>
|
|
<span style="color: #CD5555"> The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to</span>
|
|
<span style="color: #CD5555"> produce an image from a random seed. We start with a Dense layer taking this</span>
|
|
<span style="color: #CD5555"> random sample as an input and subsequently upsample through multiple</span>
|
|
<span style="color: #CD5555"> convolutional layers.</span>
|
|
<span style="color: #CD5555"> """</span>
|
|
|
|
<span style="color: #228B22"># we define our model</span>
|
|
model = tf.keras.Sequential()
|
|
|
|
|
|
<span style="color: #228B22"># adding our input layer. Dense means that every neuron is connected and</span>
|
|
<span style="color: #228B22"># the input shape is the shape of our random noise. The units need to match</span>
|
|
<span style="color: #228B22"># in some sense the upsampling strides to reach our desired output shape.</span>
|
|
<span style="color: #228B22"># we are using 100 random numbers as our seed</span>
|
|
model.add(layers.Dense(units=<span style="color: #B452CD">7</span>*<span style="color: #B452CD">7</span>*BATCH_SIZE,
|
|
use_bias=<span style="color: #8B008B; font-weight: bold">False</span>,
|
|
input_shape=(<span style="color: #B452CD">100</span>, )))
|
|
<span style="color: #228B22"># we normalize the output form the Dense layer</span>
|
|
model.add(layers.BatchNormalization())
|
|
<span style="color: #228B22"># and add an activation function to our 'layer'. LeakyReLU avoids vanishing</span>
|
|
<span style="color: #228B22"># gradient problem</span>
|
|
model.add(layers.LeakyReLU())
|
|
model.add(layers.Reshape((<span style="color: #B452CD">7</span>, <span style="color: #B452CD">7</span>, BATCH_SIZE)))
|
|
<span style="color: #8B008B; font-weight: bold">assert</span> model.output_shape == (<span style="color: #8B008B; font-weight: bold">None</span>, <span style="color: #B452CD">7</span>, <span style="color: #B452CD">7</span>, BATCH_SIZE)
|
|
<span style="color: #228B22"># even though we just added four keras layers we think of everything above</span>
|
|
<span style="color: #228B22"># as 'one' layer</span>
|
|
|
|
<span style="color: #228B22"># next we add our upscaling convolutional layers</span>
|
|
model.add(layers.Conv2DTranspose(filters=<span style="color: #B452CD">128</span>,
|
|
kernel_size=(<span style="color: #B452CD">5</span>, <span style="color: #B452CD">5</span>),
|
|
strides=(<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>),
|
|
padding=<span style="color: #CD5555">'same'</span>,
|
|
use_bias=<span style="color: #8B008B; font-weight: bold">False</span>))
|
|
model.add(layers.BatchNormalization())
|
|
model.add(layers.LeakyReLU())
|
|
<span style="color: #8B008B; font-weight: bold">assert</span> model.output_shape == (<span style="color: #8B008B; font-weight: bold">None</span>, <span style="color: #B452CD">7</span>, <span style="color: #B452CD">7</span>, <span style="color: #B452CD">128</span>)
|
|
|
|
model.add(layers.Conv2DTranspose(filters=<span style="color: #B452CD">64</span>,
|
|
kernel_size=(<span style="color: #B452CD">5</span>, <span style="color: #B452CD">5</span>),
|
|
strides=(<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2</span>),
|
|
padding=<span style="color: #CD5555">'same'</span>,
|
|
use_bias=<span style="color: #8B008B; font-weight: bold">False</span>))
|
|
model.add(layers.BatchNormalization())
|
|
model.add(layers.LeakyReLU())
|
|
<span style="color: #8B008B; font-weight: bold">assert</span> model.output_shape == (<span style="color: #8B008B; font-weight: bold">None</span>, <span style="color: #B452CD">14</span>, <span style="color: #B452CD">14</span>, <span style="color: #B452CD">64</span>)
|
|
|
|
model.add(layers.Conv2DTranspose(filters=<span style="color: #B452CD">1</span>,
|
|
kernel_size=(<span style="color: #B452CD">5</span>, <span style="color: #B452CD">5</span>),
|
|
strides=(<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2</span>),
|
|
padding=<span style="color: #CD5555">'same'</span>,
|
|
use_bias=<span style="color: #8B008B; font-weight: bold">False</span>,
|
|
activation=<span style="color: #CD5555">'tanh'</span>))
|
|
<span style="color: #8B008B; font-weight: bold">assert</span> model.output_shape == (<span style="color: #8B008B; font-weight: bold">None</span>, <span style="color: #B452CD">28</span>, <span style="color: #B452CD">28</span>, <span style="color: #B452CD">1</span>)
|
|
|
|
<span style="color: #8B008B; font-weight: bold">return</span> model
|
|
</pre></div>
|
|
<p>
|
|
And there we have our 'simple' generator model. Now we move on to defining our
|
|
discriminator model \( d \), which is a convolutional neural network based image
|
|
classifier.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">discriminator_model</span>():
|
|
<span style="color: #CD5555">"""</span>
|
|
<span style="color: #CD5555"> The discriminator is a convolutional neural network based image classifier</span>
|
|
<span style="color: #CD5555"> """</span>
|
|
|
|
<span style="color: #228B22"># we define our model</span>
|
|
model = tf.keras.Sequential()
|
|
model.add(layers.Conv2D(filters=<span style="color: #B452CD">64</span>,
|
|
kernel_size=(<span style="color: #B452CD">5</span>, <span style="color: #B452CD">5</span>),
|
|
strides=(<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2</span>),
|
|
padding=<span style="color: #CD5555">'same'</span>,
|
|
input_shape=[<span style="color: #B452CD">28</span>, <span style="color: #B452CD">28</span>, <span style="color: #B452CD">1</span>]))
|
|
model.add(layers.LeakyReLU())
|
|
<span style="color: #228B22"># adding a dropout layer as you do in conv-nets</span>
|
|
model.add(layers.Dropout(<span style="color: #B452CD">0.3</span>))
|
|
|
|
|
|
model.add(layers.Conv2D(filters=<span style="color: #B452CD">128</span>,
|
|
kernel_size=(<span style="color: #B452CD">5</span>, <span style="color: #B452CD">5</span>),
|
|
strides=(<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2</span>),
|
|
padding=<span style="color: #CD5555">'same'</span>))
|
|
model.add(layers.LeakyReLU())
|
|
<span style="color: #228B22"># adding a dropout layer as you do in conv-nets</span>
|
|
model.add(layers.Dropout(<span style="color: #B452CD">0.3</span>))
|
|
|
|
model.add(layers.Flatten())
|
|
model.add(layers.Dense(<span style="color: #B452CD">1</span>))
|
|
|
|
<span style="color: #8B008B; font-weight: bold">return</span> model
|
|
</pre></div>
|
|
<p>
|
|
Let us take a look at our models. <b>Note</b>: double click images for bigger view.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>generator = generator_model()
|
|
plot_model(generator, show_shapes=<span style="color: #8B008B; font-weight: bold">True</span>, rankdir=<span style="color: #CD5555">'LR'</span>)
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>discriminator = discriminator_model()
|
|
plot_model(discriminator, show_shapes=<span style="color: #8B008B; font-weight: bold">True</span>, rankdir=<span style="color: #CD5555">'LR'</span>)
|
|
</pre></div>
|
|
<p>
|
|
Next we need a few helper objects we will use in training
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=<span style="color: #8B008B; font-weight: bold">True</span>)
|
|
generator_optimizer = tf.keras.optimizers.Adam(<span style="color: #B452CD">1e-4</span>)
|
|
discriminator_optimizer = tf.keras.optimizers.Adam(<span style="color: #B452CD">1e-4</span>)
|
|
</pre></div>
|
|
<p>
|
|
The first object, <em>cross_entropy</em> is our loss function and the two others are
|
|
our optimizers. Notice we use the same learning rate for both \( g \) and \( d \). This
|
|
is because they need to improve their accuracy at approximately equal speeds to
|
|
get convergence (not necessarily exactly equal). Now we define our loss
|
|
functions
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">generator_loss</span>(fake_output):
|
|
loss = cross_entropy(tf.ones_like(fake_output), fake_output)
|
|
|
|
<span style="color: #8B008B; font-weight: bold">return</span> loss
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">discriminator_loss</span>(real_output, fake_output):
|
|
real_loss = cross_entropy(tf.ones_like(real_output), real_output)
|
|
fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)
|
|
total_loss = real_loss + fake_loss
|
|
|
|
<span style="color: #8B008B; font-weight: bold">return</span> total_loss
|
|
</pre></div>
|
|
<p>
|
|
Next we define a kind of seed to help us compare the learning process over
|
|
multiple training epochs.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>noise_dimension = <span style="color: #B452CD">100</span>
|
|
n_examples_to_generate = <span style="color: #B452CD">16</span>
|
|
seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])
|
|
</pre></div>
|
|
<p>
|
|
Now we have everything we need to define our training step, which we will apply
|
|
for every step in our training loop. Notice the @tf.function flag signifying
|
|
that the function is tensorflow 'compiled'. Removing this flag doubles the
|
|
computation time.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #707a7c">@tf</span>.function
|
|
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">train_step</span>(images):
|
|
noise = tf.random.normal([BATCH_SIZE, noise_dimension])
|
|
|
|
<span style="color: #8B008B; font-weight: bold">with</span> tf.GradientTape() <span style="color: #8B008B; font-weight: bold">as</span> gen_tape, tf.GradientTape() <span style="color: #8B008B; font-weight: bold">as</span> disc_tape:
|
|
generated_images = generator(noise, training=<span style="color: #8B008B; font-weight: bold">True</span>)
|
|
|
|
real_output = discriminator(images, training=<span style="color: #8B008B; font-weight: bold">True</span>)
|
|
fake_output = discriminator(generated_images, training=<span style="color: #8B008B; font-weight: bold">True</span>)
|
|
|
|
gen_loss = generator_loss(fake_output)
|
|
disc_loss = discriminator_loss(real_output, fake_output)
|
|
|
|
gradients_of_generator = gen_tape.gradient(gen_loss,
|
|
generator.trainable_variables)
|
|
gradients_of_discriminator = disc_tape.gradient(disc_loss,
|
|
discriminator.trainable_variables)
|
|
generator_optimizer.apply_gradients(<span style="color: #658b00">zip</span>(gradients_of_generator,
|
|
generator.trainable_variables))
|
|
discriminator_optimizer.apply_gradients(<span style="color: #658b00">zip</span>(gradients_of_discriminator,
|
|
discriminator.trainable_variables))
|
|
|
|
<span style="color: #8B008B; font-weight: bold">return</span> gen_loss, disc_loss
|
|
</pre></div>
|
|
<p>
|
|
Next we define a helper function to produce an output over our training epochs
|
|
to see the predictive progression of our generator model. <b>Note</b>: I am including
|
|
this code here, but comment it out in the training loop.
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">generate_and_save_images</span>(model, epoch, test_input):
|
|
<span style="color: #228B22"># we're making inferences here</span>
|
|
predictions = model(test_input, training=<span style="color: #8B008B; font-weight: bold">False</span>)
|
|
|
|
fig = plt.figure(figsize=(<span style="color: #B452CD">4</span>, <span style="color: #B452CD">4</span>))
|
|
|
|
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(predictions.shape[<span style="color: #B452CD">0</span>]):
|
|
plt.subplot(<span style="color: #B452CD">4</span>, <span style="color: #B452CD">4</span>, i+<span style="color: #B452CD">1</span>)
|
|
plt.imshow(predictions[i, :, :, <span style="color: #B452CD">0</span>] * <span style="color: #B452CD">127.5</span> + <span style="color: #B452CD">127.5</span>, cmap=<span style="color: #CD5555">'gray'</span>)
|
|
plt.axis(<span style="color: #CD5555">'off'</span>)
|
|
|
|
plt.savefig(<span style="color: #CD5555">f'./images_from_seed_images/image_at_epoch_{</span><span style="color: #658b00">str</span>(epoch).zfill(<span style="color: #B452CD">3</span>)<span style="color: #CD5555">}.png'</span>)
|
|
plt.close()
|
|
<span style="color: #228B22">#plt.show()</span>
|
|
</pre></div>
|
|
<p>
|
|
Setting up checkpoints to periodically save our model during training so that
|
|
everything is not lost even if the program were to somehow terminate while
|
|
training.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># Setting up checkpoints to save model during training</span>
|
|
checkpoint_dir = <span style="color: #CD5555">'./training_checkpoints'</span>
|
|
checkpoint_prefix = os.path.join(checkpoint_dir, <span style="color: #CD5555">'ckpt'</span>)
|
|
checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,
|
|
discriminator_optimizer=discriminator_optimizer,
|
|
generator=generator,
|
|
discriminator=discriminator)
|
|
</pre></div>
|
|
<p>
|
|
Now we define our training loop
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">train</span>(dataset, epochs):
|
|
generator_loss_list = []
|
|
discriminator_loss_list = []
|
|
|
|
<span style="color: #8B008B; font-weight: bold">for</span> epoch <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(epochs):
|
|
start = time.time()
|
|
|
|
<span style="color: #8B008B; font-weight: bold">for</span> image_batch <span style="color: #8B008B">in</span> dataset:
|
|
gen_loss, disc_loss = train_step(image_batch)
|
|
generator_loss_list.append(gen_loss.numpy())
|
|
discriminator_loss_list.append(disc_loss.numpy())
|
|
|
|
<span style="color: #228B22">#generate_and_save_images(generator, epoch + 1, seed_images)</span>
|
|
|
|
<span style="color: #8B008B; font-weight: bold">if</span> (epoch + <span style="color: #B452CD">1</span>) % <span style="color: #B452CD">15</span> == <span style="color: #B452CD">0</span>:
|
|
checkpoint.save(file_prefix=checkpoint_prefix)
|
|
|
|
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f'Time for epoch {</span>epoch<span style="color: #CD5555">} is {</span>time.time() - start<span style="color: #CD5555">}'</span>)
|
|
|
|
<span style="color: #228B22">#generate_and_save_images(generator, epochs, seed_images)</span>
|
|
|
|
loss_file = <span style="color: #CD5555">'./data/lossfile.txt'</span>
|
|
<span style="color: #8B008B; font-weight: bold">with</span> <span style="color: #658b00">open</span>(loss_file, <span style="color: #CD5555">'w'</span>) <span style="color: #8B008B; font-weight: bold">as</span> outfile:
|
|
outfile.write(<span style="color: #658b00">str</span>(generator_loss_list))
|
|
outfile.write(<span style="color: #CD5555">'\n'</span>)
|
|
outfile.write(<span style="color: #CD5555">'\n'</span>)
|
|
outfile.write(<span style="color: #658b00">str</span>(discriminator_loss_list))
|
|
outfile.write(<span style="color: #CD5555">'\n'</span>)
|
|
outfile.write(<span style="color: #CD5555">'\n'</span>)
|
|
</pre></div>
|
|
<p>
|
|
To train simply call this function. <b>Warning</b>: this might take a long time so
|
|
there is a folder of a pretrained network already included in the repository.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>train(train_dataset, EPOCHS)
|
|
</pre></div>
|
|
<p>
|
|
And here is the result of training our model for 100 epochs
|
|
|
|
<p>
|
|
|
|
<embed src="images_from_seed_images/generation.gif" autoplay="false" loop="true"></embed>
|
|
<p><em></em></p>
|
|
|
|
<p>
|
|
Now to avoid having to train and everything, which will take a while depending
|
|
on your computer setup we now load in the model which produced the above gif.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))
|
|
restored_generator = checkpoint.generator
|
|
restored_discriminator = checkpoint.discriminator
|
|
|
|
<span style="color: #658b00">print</span>(restored_generator)
|
|
<span style="color: #658b00">print</span>(restored_discriminator)
|
|
</pre></div>
|
|
|
|
<h2 id="exploring-the-latent-space">Exploring the Latent Space </h2>
|
|
|
|
<p>
|
|
So we have successfully loaded in our latest model. Let us now play around a bit
|
|
and see what kind of things we can learn about this model. Our generator takes
|
|
an array of 100 numbers. One idea can be to try to systematically change our
|
|
input. Let us try and see what we get
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">generate_latent_points</span>(number=<span style="color: #B452CD">100</span>, scale_means=<span style="color: #B452CD">1</span>, scale_stds=<span style="color: #B452CD">1</span>):
|
|
latent_dim = <span style="color: #B452CD">100</span>
|
|
means = scale_means * tf.linspace(-<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, num=latent_dim)
|
|
stds = scale_stds * tf.linspace(-<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, num=latent_dim)
|
|
latent_space_value_range = tf.random.normal([number, number],
|
|
means,
|
|
stds,
|
|
dtype=tf.float64)
|
|
|
|
<span style="color: #8B008B; font-weight: bold">return</span> latent_space_value_range
|
|
|
|
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">generate_images</span>(latent_points):
|
|
<span style="color: #228B22"># notice we set training to false because we are making inferences</span>
|
|
generated_images = restored_generator(latent_space_value_range,
|
|
training=<span style="color: #8B008B; font-weight: bold">False</span>)
|
|
|
|
<span style="color: #8B008B; font-weight: bold">return</span> generated_images
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">plot_result</span>(generated_images, number):
|
|
<span style="color: #228B22"># obviously this assumes sqrt number is an int</span>
|
|
fig, axs = plt.subplots(<span style="color: #658b00">int</span>(np.sqrt(number)), <span style="color: #658b00">int</span>(np.sqrt(number)),
|
|
figsize=(<span style="color: #B452CD">10</span>, <span style="color: #B452CD">10</span>))
|
|
|
|
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">int</span>(np.sqrt(number))):
|
|
<span style="color: #8B008B; font-weight: bold">for</span> j <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">int</span>(np.sqrt(number))):
|
|
axs[i, j].imshow(generated_images[i*j], cmap=<span style="color: #CD5555">'Greys'</span>)
|
|
axs[i, j].axis(<span style="color: #CD5555">'off'</span>)
|
|
|
|
plt.show()
|
|
</pre></div>
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>generated_images = generate_images(generate_latent_points())
|
|
plot_result(generated_images, number)
|
|
</pre></div>
|
|
<p>
|
|
Interesting! We see that the generator generates images that look like MNIST
|
|
numbers: \( 1, 4, 7, 9 \). Let's try to tweak it a bit more to see if we are able
|
|
to generate a similar plot where we generate every MNIST number. Let us now try
|
|
to 'move' a bit around in the latent space. <b>Note</b>: decrease the plot number if
|
|
these following cells take too long to run on your computer.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>plot_number = <span style="color: #B452CD">225</span>
|
|
|
|
generated_images = generate_images(generate_latent_points(number=plot_number,
|
|
scale_means=<span style="color: #B452CD">5</span>,
|
|
scale_stds=<span style="color: #B452CD">1</span>))
|
|
plot_result(generated_images, plot_number)
|
|
|
|
generated_images = generate_images(generate_latent_points(number=plot_number,
|
|
scale_means=-<span style="color: #B452CD">5</span>,
|
|
scale_stds=<span style="color: #B452CD">1</span>))
|
|
plot_result(generated_images, plot_number)
|
|
|
|
generated_images = generate_images(generate_latent_points(number=plot_number,
|
|
scale_means=<span style="color: #B452CD">1</span>,
|
|
scale_stds=<span style="color: #B452CD">5</span>))
|
|
plot_result(generated_images, plot_number)
|
|
</pre></div>
|
|
<p>
|
|
Again, we have found something interesting. <em>Moving</em> around using our means
|
|
takes us from digit to digit, while <em>moving</em> around using our standard
|
|
deviations seem to increase the number of different digits! In the last image
|
|
above, we can barely make out every MNIST digit. Let us make on last plot using
|
|
this information by upping the standard deviation of our Gaussian noises.
|
|
|
|
<p>
|
|
|
|
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
|
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>plot_number = <span style="color: #B452CD">400</span>
|
|
generated_images = generate_images(generate_latent_points(number=plot_number,
|
|
scale_means=<span style="color: #B452CD">1</span>,
|
|
scale_stds=<span style="color: #B452CD">10</span>))
|
|
</pre></div>
|
|
<p>
|
|
A pretty cool result! We see that our generator indeed has learned a
|
|
distribution which qualitatively looks a whole lot like the MNIST dataset.
|
|
|
|
<h2 id="interpolating-between-mnist-digits">Interpolating Between MNIST Digits </h2>
|
|
Another interesting way to explore the latent space of our generator model is by
|
|
interpolating between the MNIST digits. This section is largely based on
|
|
"this excellent blogpost": <a href="https://machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/" target="_blank"><tt>https://machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/</tt></a>
|
|
by Jason Brownlee.
|
|
|
|
<p>
|
|
So let us start
|
|
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|