lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
02bca1aa1aebd47407af074a8dcad5fc51fef9f4
FYS-STK4155/doc/LectureNotes/_build/html/Clustering.html
T
Morten Hjorth-Jensen 6c32c5b5e2 tensorflow problems with chapter 12
2021-10-26 22:24:12 +02:00

1040 lines
79 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>12. Clustering Analysis &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet" />
<link href="_static/css/index.c5995385ac14fb8791e8eb36b4908be2.css" rel="stylesheet" />
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/sphinx-book-theme.5f77b4aec8189eecf79907ce328c390d.css" type="text/css" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
<link rel="preload" as="script" href="_static/js/index.1c5a1a01449ed65a7b51.js">
<script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/language_data.js"></script>
<script src="_static/togglebutton.js"></script>
<script src="_static/clipboard.min.js"></script>
<script src="_static/copybutton.js"></script>
<script >var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
<script src="_static/sphinx-book-theme.12a9622fbb08dcb3a2a40b2c02b83a57.js"></script>
<script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/x-mathjax-config">MathJax.Hub.Config({"tex2jax": {"inlineMath": [["\\(", "\\)"]], "displayMath": [["\\[", "\\]"]], "processRefs": false, "processEnvironments": false}})</script>
<script async="async" src="https://unpkg.com/thebelab@latest/lib/index.js"></script>
<script >
const thebe_selector = ".thebe"
const thebe_selector_input = "pre"
const thebe_selector_output = ".output"
</script>
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="13. Neural networks" href="chapter9.html" />
<link rel="prev" title="11. Basic ideas of the Principal Component Analysis (PCA)" href="chapter8.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
<div class="container-fluid" id="banner"></div>
<div class="container-xl">
<div class="row">
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
<div class="navbar-brand-box">
<a class="navbar-brand text-wrap" href="index.html">
<img src="_static/logo.png" class="logo" alt="logo">
<h1 class="site-logo" id="site-title">Applied Data Analysis and Machine Learning</h1>
</a>
</div><form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Applied Data Analysis and Machine Learning, FYS-STK3155/4155 at the University of Oslo, Norway
</a>
</li>
</ul>
<p class="caption">
<span class="caption-text">
About the course
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="schedule.html">
Teaching schedule with links to material
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="teachers.html">
Teachers and Grading
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="textbooks.html">
Textbooks
</a>
</li>
</ul>
<p class="caption">
<span class="caption-text">
Review of Statistics with Resampling Techniques and Linear Algebra
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="statistics.html">
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="linalg.html">
2. Linear Algebra, Handling of Arrays and more Python Features
</a>
</li>
</ul>
<p class="caption">
<span class="caption-text">
From Regression to Support Vector Machines
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html">
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
6. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapteroptimization.html">
7. Optimization, the central part of any Machine Learning algortithm
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
<p class="caption">
<span class="caption-text">
Decision Trees, Ensemble Methods and Boosting
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
<p class="caption">
<span class="caption-text">
Dimensionality Reduction
</span>
</p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
12. Clustering Analysis
</a>
</li>
</ul>
<p class="caption">
<span class="caption-text">
Deep Learning Methods
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
14. Building a Feed Forward Neural Network
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter11.html">
15. Solving Differential Equations with Deep Learning
</a>
</li>
</ul>
</div>
</nav> <!-- To handle the deprecated key -->
<div class="navbar_extra_footer">
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
</div>
</div>
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
<div class="topbar container-xl fixed-top">
<div class="topbar-contents row">
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show"></div>
<div class="col pl-md-4 topbar-main">
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
<i class="fas fa-bars"></i>
<i class="fas fa-arrow-left"></i>
<i class="fas fa-arrow-up"></i>
</button>
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
class="fas fa-download"></i></button>
<div class="dropdown-buttons">
<!-- ipynb file if we had a myst markdown file -->
<!-- Download raw file -->
<a class="dropdown-buttons" href="_sources/Clustering.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
data-placement="left">.ipynb</button></a>
<!-- Download PDF via print -->
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
onClick="window.print()" data-toggle="tooltip" data-placement="left">.pdf</button>
</div>
</div>
<!-- Source interaction buttons -->
<!-- Full screen (wrap in <a> to have style consistency -->
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
data-placement="bottom" onclick="toggleFullScreen()" aria-label="Fullscreen mode"
title="Fullscreen mode"><i
class="fas fa-expand"></i></button></a>
<!-- Launch buttons -->
</div>
<!-- Table of contents -->
<div class="d-none d-md-block col-md-2 bd-toc show">
<div class="tocsection onthispage pt-5 pb-3">
<i class="fas fa-list"></i> Contents
</div>
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-idea-of-the-k-means-clustering-algorithm">
12.1. Basic Idea of the K-means Clustering Algorithm
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-k-means-clustering-algorithm">
12.2. The K-means Clustering Algorithm
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#writing-our-own-code">
12.3. Writing Our Own Code
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-python">
12.3.1. Basic Python
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#towards-a-more-numpythonic-code">
12.4. Towards a More Numpythonic Code
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div id="main-content" class="row">
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
<div>
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html Clustering.do.txt --><div class="section" id="clustering-analysis">
<h1><span class="section-number">12. </span>Clustering Analysis<a class="headerlink" href="#clustering-analysis" title="Permalink to this headline"></a></h1>
<p>In this chapter we will concern ourselves with the study of <strong>cluster analysis</strong>.
In general terms cluster analysis, or clustering, is the task of grouping a
data-set into different distinct categories based on some measure of equality of
the data. This measure is often referred to as a <strong>metric</strong> or <strong>similarity
measure</strong> in the literature (note: sometimes we deal with a <strong>dissimilarity
measure</strong> instead). Usually, these metrics are formulated as some kind of
distance function between points in a high-dimensional space.</p>
<p>There exists a lot of such distance measures. The simplest, and also the most
common is the <strong>Euclidean distance</strong> (i.e. Pythagoras). A good source for those of
you wanting a thorough overview is the article (DOI:10.5120/ijca2016907841
Irani, Pise, Phatak). A few other metrics mentioned there are: <em>cosine
similarity</em>, <em>Manhattan distance</em>, <em>Chebychev distance</em> and the <em>Minkowski
distance</em>. The Minkowski distance is a general formulation which encapsulates a
range of metrics. All of these, and many more, can be used in clustering. There
exists different categories of clustering algorithms. A few of the most
common are: <em>centroid-</em>, <em>distribution-</em>, <em>density-</em> and <em>hierarchical-
clustering</em>. We will concern ourselves primarily with the first one.</p>
<div class="section" id="basic-idea-of-the-k-means-clustering-algorithm">
<h2><span class="section-number">12.1. </span>Basic Idea of the K-means Clustering Algorithm<a class="headerlink" href="#basic-idea-of-the-k-means-clustering-algorithm" title="Permalink to this headline"></a></h2>
<p>The simplest of all clustering algorithms is the aptly named <strong>k-means algorithm</strong>
, sometimes also referred to as <em>Lloyds algorithm</em>. It is the simplest and also
the most common. From its simplicity it obtains both strengths and weaknesses.
These will be discussed in more detail later. The k-means algorithm is a
<strong>centroid based</strong> clustering algorithm.</p>
<p>Assume, we are given <span class="math notranslate nohighlight">\(n\)</span> data points and we wish to split the data into <span class="math notranslate nohighlight">\(K &lt; n\)</span>
different categories, or clusters. We label each cluster by an integer <span class="math notranslate nohighlight">\(k\in\{
1, \cdots, K \}\)</span>. In the basic k-means algorithm each point is assigned to only
one cluster <span class="math notranslate nohighlight">\(k\)</span>, and these assignments are <em>non-injective</em> i.e. many-to-one. We
can think of these mappings as an encoder <span class="math notranslate nohighlight">\(k = C(i)\)</span>, which assigns the <span class="math notranslate nohighlight">\(i\)</span>-th
data-point <span class="math notranslate nohighlight">\(\bf x_i\)</span> to the <span class="math notranslate nohighlight">\(k\)</span>-th cluster. Before we jump into the mathematics
let us describe the k-means algorithm in words:</p>
<ol class="simple">
<li><p>We start with guesses / random initializations of our <span class="math notranslate nohighlight">\(k\)</span> cluster centers / centroids</p></li>
<li><p>For each centroid the points that are most similar are identified</p></li>
<li><p>Then we move / replace each centroid with a coordinate average of all the points that were assigned to that centroid.</p></li>
<li><p>Iterate this points 2, 3) until the centroids no longer move (to some tolerance)</p></li>
</ol>
<p>Now we consider the method formally. Again, we assume we have <span class="math notranslate nohighlight">\(n\)</span> data-points
(vectors)</p>
<!-- Equation labels as ordinary links -->
<div id="eq:kmeanspoints"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}\label{eq:kmeanspoints} \tag{1}
\boldsymbol{x_i} = \{x_{i, 1}, \cdots, x_{i, p}\}\in\mathbb{R}^p.
\end{equation}
\]</div>
<p>which we wish to group into <span class="math notranslate nohighlight">\(K &lt; n\)</span> clusters. For our dissimilarity measure we
will use the <em>squared Euclidean distance</em></p>
<!-- Equation labels as ordinary links -->
<div id="eq:squaredeuclidean"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}\label{eq:squaredeuclidean} \tag{2}
d(\boldsymbol{x_i}, \boldsymbol{x_i'}) = \sum_{j=1}^p(x_{ij} - x_{i'j})^2
= ||\boldsymbol{x_i} - \boldsymbol{x_{i'}}||^2
\end{equation}
\]</div>
<p>Next we define the so called <em>within-cluster point scatter</em> which gives us a
measure of how close each data point assigned to the same cluster tends to be to
the all the others.</p>
<!-- Equation labels as ordinary links -->
<div id="eq:withincluster"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}\label{eq:withincluster} \tag{3}
W(C) = \frac{1}{2}\sum_{k=1}^K\sum_{C(i)=k}
\sum_{C(i')=k}d(\boldsymbol{x_i}, \boldsymbol{x_{i'}}) =
\sum_{k=1}^KN_k\sum_{C(i)=k}||\boldsymbol{x_i} - \boldsymbol{\overline{x_k}}||^2
\end{equation}
\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\overline{x_k}}\)</span> is the mean vector associated with the <span class="math notranslate nohighlight">\(k\)</span>-th
cluster, and <span class="math notranslate nohighlight">\(N_k = \sum_{i=1}^nI(C(i) = k)\)</span>, where the <span class="math notranslate nohighlight">\(I()\)</span> notation is
similar to the Kronecker delta (<em>Commonly used in statistics, it just means that
when <span class="math notranslate nohighlight">\(i = k\)</span> we have the encoder <span class="math notranslate nohighlight">\(C(i)\)</span></em>). In other words, the within-cluster
scatter measures the compactness of each cluster with respect to the data points
assigned to each cluster. This is the quantity that the <span class="math notranslate nohighlight">\(k\)</span>-means algorithm aims
to minimize. We refer to this quantity <span class="math notranslate nohighlight">\(W(C)\)</span> as the within cluster scatter
because of its relation to the <em>total scatter</em>.</p>
<!-- Equation labels as ordinary links -->
<div id="eq:totalscatter"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}\label{eq:totalscatter} \tag{4}
T = W(C) + B(C) = \frac{1}{2}\sum_{i=1}^n
\sum_{i'=1}^nd(\boldsymbol{x_i}, \boldsymbol{x_{i'}})
= \frac{1}{2}\sum_{k=1}^K\sum_{C(i)=k}
\Big(\sum_{C(i') = k}d(\boldsymbol{x_i}, \boldsymbol{x_{i'}})
+ \sum_{C(i')\neq k}d(\boldsymbol{x_i}, \boldsymbol{x_{i'}})\Big)
\end{equation}
\]</div>
<p>Which is a quantity that is conserved throughout the <span class="math notranslate nohighlight">\(k\)</span>-means algorithm. It can
be thought of as the total amount of information in the data, and it is composed
of the aforementioned within-cluster scatter and the <em>between-cluster scatter</em>
<span class="math notranslate nohighlight">\(B(C)\)</span>. In methods such as principle component analysis the total scatter is not
conserved.</p>
<p>Given a cluster mean <span class="math notranslate nohighlight">\(\boldsymbol{m_k}\)</span> we define the <strong>total cluster variance</strong></p>
<!-- Equation labels as ordinary links -->
<div id="eq:totalclustervariance"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}\label{eq:totalclustervariance} \tag{5}
\min_{C, \{\boldsymbol{m_k}\}_1^K}\sum_{k=1}^KN_k\sum||\boldsymbol{x_i} - \boldsymbol{m_k}||^2
\end{equation}
\]</div>
<p>Now we have all the pieces necessary to formally revisit the k-means algorithm.
If you at this point feel like some of the above definitions came a bit out of
no-where, don’t fret, the method does get a whole lot simpler once we start
programming.</p>
</div>
<div class="section" id="the-k-means-clustering-algorithm">
<h2><span class="section-number">12.2. </span>The K-means Clustering Algorithm<a class="headerlink" href="#the-k-means-clustering-algorithm" title="Permalink to this headline"></a></h2>
<p>The k-means clustering algorithm goes as follows (note in my opinion this
description is a bit complicated and is lifted directly out of ESL HASTIE for
deeper understanding purposes)</p>
<ol class="simple">
<li><p>For a given cluster assignment <span class="math notranslate nohighlight">\(C\)</span>, and <span class="math notranslate nohighlight">\(k\)</span> cluster means <span class="math notranslate nohighlight">\(\{m_1, \cdots, m_k\}\)</span>. We minimize the total cluster variance with respect to the cluster means <span class="math notranslate nohighlight">\(\{m_k\}\)</span> yielding the means of the currently assigned clusters.</p></li>
<li><p>Given a current set of <span class="math notranslate nohighlight">\(k\)</span> means <span class="math notranslate nohighlight">\(\{m_k\}\)</span> the total cluster variance is minimized by assigning each observation to the closest (current) cluster mean. That is $<span class="math notranslate nohighlight">\(C(i) = \underset{1\leq k\leq K}{\mathrm{argmin}} ||\boldsymbol{x_i} - \boldsymbol{m_k}||^2\)</span>$</p></li>
<li><p>Steps 1 and 2 are repeated until the assignments do not change.</p></li>
</ol>
<p>As previously stated the above formulation can be a bit difficult to understand,
<em>at least the first time</em>, due to the dense notation used. But all in all the
concept is fairly simple when explained in words. The math needs to be
understood but to help you along the way we summarize the algorithm as follows
(try to look at the terms above to match with the summary).</p>
<ol class="simple">
<li><p>Before we start we specify a number <span class="math notranslate nohighlight">\(k\)</span> which is the number of clusters we want to try to separate our data into.</p></li>
<li><p>We initially choose <span class="math notranslate nohighlight">\(k\)</span> random data points in our data as our initial centroids, <em>or means</em> (this is where the name comes from).</p></li>
<li><p>Assign each data point to their closest centroid, based on the squared Euclidean distance.</p></li>
<li><p>For each of the <span class="math notranslate nohighlight">\(k\)</span> cluster we update the centroid by calculating new mean values for all the data points in the cluster.</p></li>
<li><p>Iteratively minimize the within cluster scatter by performing steps (3, 4) until the new assignments stop changing (can be to some tolerance) or until a maximum number of iterations have passed.</p></li>
</ol>
<p>That’s it, nothing magical happening.</p>
</div>
<div class="section" id="writing-our-own-code">
<h2><span class="section-number">12.3. </span>Writing Our Own Code<a class="headerlink" href="#writing-our-own-code" title="Permalink to this headline"></a></h2>
<p>In the following section we will work to develop a deeper understanding of the
previously discussed mathematics through developing codes to do k-means cluster
analysis.</p>
<div class="section" id="basic-python">
<h3><span class="section-number">12.3.1. </span>Basic Python<a class="headerlink" href="#basic-python" title="Permalink to this headline"></a></h3>
<p>Let us now program the most basic version of the algorithm using nothing but
Python with numpy arrays. This code is kept intentionally simple to gradually
progress our understanding. There is no vectorization of any kind, and even most
helper functions are not utilized. Throughout our implementation process it will
be helpful to keep in mind both the mathematical description of the algorithm
<em>and</em> our summary from above. In addition, try to think of ways to optimize this
while reading the next section. We will get to it, take it as a challenge to see
if your optimizations are better.</p>
<p>First of all we need a dataset to do our cluster analysis on, for clarity (and
lack of googling beforehand) we generate it ourselves using Gaussians. First we
import</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
<span class="o">%</span><span class="k">matplotlib</span> inline
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">display</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2021</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>Next we define functions, for ease of use later, to generate Gaussians and to
set up our toy data set.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">gaussian_points</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">n_points</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">mean_vector</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">]),</span>
<span class="n">sample_variance</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Very simple custom function to generate gaussian distributed point clusters</span>
<span class="sd"> with variable dimension, number of points, means in each direction</span>
<span class="sd"> (must match dim) and sample variance.</span>
<span class="sd"> Inputs:</span>
<span class="sd"> dim (int)</span>
<span class="sd"> n_points (int)</span>
<span class="sd"> mean_vector (np.array) (where index 0 is x, index 1 is y etc.)</span>
<span class="sd"> sample_variance (float)</span>
<span class="sd"> Returns:</span>
<span class="sd"> data (np.array): with dimensions (dim x n_points)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">mean_matrix</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">dim</span><span class="p">)</span> <span class="o">+</span> <span class="n">mean_vector</span>
<span class="n">covariance_matrix</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">eye</span><span class="p">(</span><span class="n">dim</span><span class="p">)</span> <span class="o">*</span> <span class="n">sample_variance</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">multivariate_normal</span><span class="p">(</span><span class="n">mean_matrix</span><span class="p">,</span> <span class="n">covariance_matrix</span><span class="p">,</span>
<span class="n">n_points</span><span class="p">)</span>
<span class="k">return</span> <span class="n">data</span>
<span class="k">def</span> <span class="nf">generate_simple_clustering_dataset</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">n_points</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">plotting</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">return_data</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Toy model to illustrate k-means clustering</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">data1</span> <span class="o">=</span> <span class="n">gaussian_points</span><span class="p">(</span><span class="n">mean_vector</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">]))</span>
<span class="n">data2</span> <span class="o">=</span> <span class="n">gaussian_points</span><span class="p">()</span>
<span class="n">data3</span> <span class="o">=</span> <span class="n">gaussian_points</span><span class="p">(</span><span class="n">mean_vector</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mf">4.5</span><span class="p">]))</span>
<span class="n">data4</span> <span class="o">=</span> <span class="n">gaussian_points</span><span class="p">(</span><span class="n">mean_vector</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">5</span><span class="p">,</span> <span class="mi">1</span><span class="p">]))</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">((</span><span class="n">data1</span><span class="p">,</span> <span class="n">data2</span><span class="p">,</span> <span class="n">data3</span><span class="p">,</span> <span class="n">data4</span><span class="p">),</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="k">if</span> <span class="n">plotting</span><span class="p">:</span>
<span class="n">fig</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>
<span class="n">ax</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">data</span><span class="p">[:,</span> <span class="mi">0</span><span class="p">],</span> <span class="n">data</span><span class="p">[:,</span> <span class="mi">1</span><span class="p">],</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="s1">&#39;Toy Model Dataset&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="k">if</span> <span class="n">return_data</span><span class="p">:</span>
<span class="k">return</span> <span class="n">data</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">generate_simple_clustering_dataset</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/Clustering_18_0.png" src="_images/Clustering_18_0.png" />
</div>
</div>
<p>Now that we are our, albeit very simple, dataset we are ready to start
implementing the k-means algorithm.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">n_samples</span><span class="p">,</span> <span class="n">dimensions</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">shape</span>
<span class="n">n_clusters</span> <span class="o">=</span> <span class="mi">4</span>
<span class="c1"># we randomly initialize our centroids</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2021</span><span class="p">)</span>
<span class="n">centroids</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">n_samples</span><span class="p">,</span> <span class="n">n_clusters</span><span class="p">,</span> <span class="n">replace</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="p">:]</span>
<span class="n">distances</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">n_samples</span><span class="p">,</span> <span class="n">n_clusters</span><span class="p">))</span>
<span class="c1"># first we need to calculate the distance to each centroid from our data</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">):</span>
<span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_samples</span><span class="p">):</span>
<span class="n">dist</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">dimensions</span><span class="p">):</span>
<span class="n">dist</span> <span class="o">+=</span> <span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="n">d</span><span class="p">]</span> <span class="o">-</span> <span class="n">centroids</span><span class="p">[</span><span class="n">k</span><span class="p">,</span> <span class="n">d</span><span class="p">])</span><span class="o">**</span><span class="mi">2</span>
<span class="n">distances</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">dist</span>
<span class="c1"># we initialize an array to keep track of to which cluster each point belongs</span>
<span class="c1"># the way we set it up here the index tracks which point and the value which</span>
<span class="c1"># cluster the point belongs to</span>
<span class="n">cluster_labels</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">n_samples</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">&#39;int&#39;</span><span class="p">)</span>
<span class="c1"># next we loop through our samples and for every point assign it to the cluster</span>
<span class="c1"># to which it has the smallest distance to</span>
<span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_samples</span><span class="p">):</span>
<span class="c1"># tracking variables (all of this is basically just an argmin)</span>
<span class="n">smallest</span> <span class="o">=</span> <span class="mf">1e10</span>
<span class="n">smallest_row_index</span> <span class="o">=</span> <span class="mf">1e10</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">):</span>
<span class="k">if</span> <span class="n">distances</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="n">k</span><span class="p">]</span> <span class="o">&lt;</span> <span class="n">smallest</span><span class="p">:</span>
<span class="n">smallest</span> <span class="o">=</span> <span class="n">distances</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="n">k</span><span class="p">]</span>
<span class="n">smallest_row_index</span> <span class="o">=</span> <span class="n">k</span>
<span class="n">cluster_labels</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="n">smallest_row_index</span>
</pre></div>
</div>
</div>
</div>
<p>Let’s plot and see</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_subplot</span><span class="p">()</span>
<span class="n">unique_cluster_labels</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">unique</span><span class="p">(</span><span class="n">cluster_labels</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">unique_cluster_labels</span><span class="p">:</span>
<span class="n">ax</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">cluster_labels</span> <span class="o">==</span> <span class="n">i</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span>
<span class="n">data</span><span class="p">[</span><span class="n">cluster_labels</span> <span class="o">==</span> <span class="n">i</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">i</span><span class="p">,</span>
<span class="n">alpha</span> <span class="o">=</span> <span class="mf">0.2</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">centroids</span><span class="p">[:,</span> <span class="mi">0</span><span class="p">],</span> <span class="n">centroids</span><span class="p">[:,</span> <span class="mi">1</span><span class="p">],</span> <span class="n">c</span><span class="o">=</span><span class="s1">&#39;black&#39;</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="s2">&quot;First Grouping of Points to Centroids&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/Clustering_22_0.png" src="_images/Clustering_22_0.png" />
</div>
</div>
<p>So what do we have so far? We have ‘picked’ <span class="math notranslate nohighlight">\(k\)</span> centroids at random from our
data points. There are other ways of more intelligently choosing their
initializations, however for our purposes randomly is fine. Then we have
initialized an array ‘distances’ which holds the information of the distance,
<em>or dissimilarity</em>, of every point to of our centroids. Finally, we have
initialized an array ‘cluster_labels’ which according to our distances array
holds the information of to which centroid every point is assigned. This was the
first pass of our algorithm. Essentially, all we need to do now is repeat the
distance and assignment steps above until we have reached a desired convergence
or a maximum amount of iterations.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">max_iterations</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">tolerance</span> <span class="o">=</span> <span class="mf">1e-8</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="k">for</span> <span class="n">iteration</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">max_iterations</span><span class="p">):</span>
<span class="n">prev_centroids</span> <span class="o">=</span> <span class="n">centroids</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">):</span>
<span class="c1"># this array will be used to update our centroid positions</span>
<span class="n">vector_mean</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">dimensions</span><span class="p">)</span>
<span class="n">mean_divisor</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_samples</span><span class="p">):</span>
<span class="k">if</span> <span class="n">cluster_labels</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">==</span> <span class="n">k</span><span class="p">:</span>
<span class="n">vector_mean</span> <span class="o">+=</span> <span class="n">data</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="p">:]</span>
<span class="n">mean_divisor</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="c1"># update according to the k means</span>
<span class="n">centroids</span><span class="p">[</span><span class="n">k</span><span class="p">,</span> <span class="p">:]</span> <span class="o">=</span> <span class="n">vector_mean</span> <span class="o">/</span> <span class="n">mean_divisor</span>
<span class="c1"># we find the dissimilarity</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">):</span>
<span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_samples</span><span class="p">):</span>
<span class="n">dist</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">dimensions</span><span class="p">):</span>
<span class="n">dist</span> <span class="o">+=</span> <span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="n">d</span><span class="p">]</span> <span class="o">-</span> <span class="n">centroids</span><span class="p">[</span><span class="n">k</span><span class="p">,</span> <span class="n">d</span><span class="p">])</span><span class="o">**</span><span class="mi">2</span>
<span class="n">distances</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">dist</span>
<span class="c1"># assign each point</span>
<span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_samples</span><span class="p">):</span>
<span class="n">smallest</span> <span class="o">=</span> <span class="mf">1e10</span>
<span class="n">smallest_row_index</span> <span class="o">=</span> <span class="mf">1e10</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">):</span>
<span class="k">if</span> <span class="n">distances</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="n">k</span><span class="p">]</span> <span class="o">&lt;</span> <span class="n">smallest</span><span class="p">:</span>
<span class="n">smallest</span> <span class="o">=</span> <span class="n">distances</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="n">k</span><span class="p">]</span>
<span class="n">smallest_row_index</span> <span class="o">=</span> <span class="n">k</span>
<span class="n">cluster_labels</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="n">smallest_row_index</span>
<span class="c1"># convergence criteria</span>
<span class="n">centroid_difference</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">centroids</span> <span class="o">-</span> <span class="n">prev_centroids</span><span class="p">))</span>
<span class="k">if</span> <span class="n">centroid_difference</span> <span class="o">&lt;</span> <span class="n">tolerance</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Converged at iteration </span><span class="si">{</span><span class="n">iteration</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Runtime: </span><span class="si">{</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_time</span><span class="si">}</span><span class="s1"> seconds&#39;</span><span class="p">)</span>
<span class="k">break</span>
<span class="k">elif</span> <span class="n">iteration</span> <span class="o">==</span> <span class="n">max_iterations</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Did not converge in </span><span class="si">{</span><span class="n">max_iterations</span><span class="si">}</span><span class="s1"> iterations&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Runtime: </span><span class="si">{</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_time</span><span class="si">}</span><span class="s1"> seconds&#39;</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Converged at iteration 5
Runtime: 0.4887218475341797 seconds
</pre></div>
</div>
</div>
</div>
<p>And thats it! We now have an extremely barebones, un-optimized k-means
clustering implementation. Lets plot the final result</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_subplot</span><span class="p">()</span>
<span class="n">unique_cluster_labels</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">unique</span><span class="p">(</span><span class="n">cluster_labels</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">unique_cluster_labels</span><span class="p">:</span>
<span class="n">ax</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">cluster_labels</span> <span class="o">==</span> <span class="n">i</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span>
<span class="n">data</span><span class="p">[</span><span class="n">cluster_labels</span> <span class="o">==</span> <span class="n">i</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">i</span><span class="p">,</span>
<span class="n">alpha</span> <span class="o">=</span> <span class="mf">0.2</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">centroids</span><span class="p">[:,</span> <span class="mi">0</span><span class="p">],</span> <span class="n">centroids</span><span class="p">[:,</span> <span class="mi">1</span><span class="p">],</span> <span class="n">c</span><span class="o">=</span><span class="s1">&#39;black&#39;</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="s2">&quot;Final Result of K-means Clustering&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/Clustering_26_0.png" src="_images/Clustering_26_0.png" />
</div>
</div>
<p>Now there are a few glaring improvements to be done here. First of all is
organizing things into functions for better readability. Second is getting rid
of the small inefficiencies like manually calculating distances and argmin. And
finally, we need to optimize for better run-time. It’s like we always say: the
best way of looping in Python is to not loop in Python. Let us tackle the first
two improvements.</p>
</div>
</div>
<div class="section" id="towards-a-more-numpythonic-code">
<h2><span class="section-number">12.4. </span>Towards a More Numpythonic Code<a class="headerlink" href="#towards-a-more-numpythonic-code" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_distances_to_clusters</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">centroids</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Function that for each cluster finds the squared Euclidean distance</span>
<span class="sd"> from every data point to the cluster centroid and returns a numpy array</span>
<span class="sd"> containing the distances such that distance[i, j] means the distance between</span>
<span class="sd"> the i-th point and the j-th centroid.</span>
<span class="sd"> Inputs:</span>
<span class="sd"> data (np.array): with dimensions (n_samples x dim)</span>
<span class="sd"> centroids (np.array): with dimensions (n_clusters x dim)</span>
<span class="sd"> Returns:</span>
<span class="sd"> distances (np.array): with dimensions (n_samples x n_clusters)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">n_samples</span><span class="p">,</span> <span class="n">dimensions</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">shape</span>
<span class="n">n_clusters</span> <span class="o">=</span> <span class="n">centroids</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">distances</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">n_samples</span><span class="p">,</span> <span class="n">n_clusters</span><span class="p">))</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_samples</span><span class="p">):</span>
<span class="n">dist</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">dimensions</span><span class="p">):</span>
<span class="n">dist</span> <span class="o">+=</span> <span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">]</span> <span class="o">-</span> <span class="n">centroids</span><span class="p">[</span><span class="n">k</span><span class="p">,</span> <span class="n">j</span><span class="p">])</span><span class="o">**</span><span class="mi">2</span>
<span class="n">distances</span><span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">dist</span>
<span class="k">return</span> <span class="n">distances</span>
<span class="k">def</span> <span class="nf">assign_points_to_clusters</span><span class="p">(</span><span class="n">distances</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Function to assign each data point to the cluster to which it is the closest</span>
<span class="sd"> based on the squared Euclidean distance from the get_distances_to_clusters</span>
<span class="sd"> method.</span>
<span class="sd"> Inputs:</span>
<span class="sd"> distances (np.array): with dimensions (n_samples x n_clusters)</span>
<span class="sd"> Returns:</span>
<span class="sd"> cluster_labels (np.array): with dimensions (n_samples)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">cluster_labels</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">argmin</span><span class="p">(</span><span class="n">distances</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="k">return</span> <span class="n">cluster_labels</span>
<span class="k">def</span> <span class="nf">k_means</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">n_clusters</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">max_iterations</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">tolerance</span><span class="o">=</span><span class="mf">1e-8</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Naive implementation of the k-means clustering algorithm. A short summary of</span>
<span class="sd"> the algorithm is as follows: we randomly initialize k centroids / means.</span>
<span class="sd"> Then we assign, using the squared Euclidean distance, every data-point to a</span>
<span class="sd"> cluster. We then update the position of the k centroids / means, and repeat</span>
<span class="sd"> until convergence or we reach our desired maximum iterations. The method</span>
<span class="sd"> returns the cluster assignments of our data-points and a sequence of</span>
<span class="sd"> centroids.</span>
<span class="sd"> Inputs:</span>
<span class="sd"> data (np.array): with dimesions (n_samples x dim)</span>
<span class="sd"> n_clusters (int): hyperparameter which depends on dataset</span>
<span class="sd"> max_iterations (int): hyperparameter which depends on dataset</span>
<span class="sd"> tolerance (float): convergence measure</span>
<span class="sd"> Returns:</span>
<span class="sd"> cluster_labels (np.array): with dimension (n_samples)</span>
<span class="sd"> centroid_list (list): list of centroids (np.array)</span>
<span class="sd"> with dimensions (n_clusters x dim)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">samples</span><span class="p">,</span> <span class="n">dimensions</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">shape</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2021</span><span class="p">)</span>
<span class="n">centroids</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="n">n_clusters</span><span class="p">,</span> <span class="n">replace</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="p">:]</span>
<span class="n">distances</span> <span class="o">=</span> <span class="n">get_distances_to_clusters</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">centroids</span><span class="p">)</span>
<span class="n">cluster_labels</span> <span class="o">=</span> <span class="n">assign_points_to_clusters</span><span class="p">(</span><span class="n">distances</span><span class="p">)</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="k">for</span> <span class="n">iteration</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">max_iterations</span><span class="p">):</span>
<span class="n">prev_centroids</span> <span class="o">=</span> <span class="n">centroids</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">):</span>
<span class="n">vector_mean</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">dimensions</span><span class="p">)</span>
<span class="n">mean_divisor</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_samples</span><span class="p">):</span>
<span class="k">if</span> <span class="n">cluster_labels</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">==</span> <span class="n">k</span><span class="p">:</span>
<span class="n">vector_mean</span> <span class="o">+=</span> <span class="n">data</span><span class="p">[</span><span class="n">n</span><span class="p">,</span> <span class="p">:]</span>
<span class="n">mean_divisor</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="c1"># And update according to the new means</span>
<span class="n">centroids</span><span class="p">[</span><span class="n">k</span><span class="p">,</span> <span class="p">:]</span> <span class="o">=</span> <span class="n">vector_mean</span> <span class="o">/</span> <span class="n">mean_divisor</span>
<span class="n">distances</span> <span class="o">=</span> <span class="n">get_distances_to_clusters</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">centroids</span><span class="p">)</span>
<span class="n">cluster_labels</span> <span class="o">=</span> <span class="n">assign_points_to_clusters</span><span class="p">(</span><span class="n">distances</span><span class="p">)</span>
<span class="n">centroid_difference</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">centroids</span> <span class="o">-</span> <span class="n">prev_centroids</span><span class="p">))</span>
<span class="k">if</span> <span class="n">centroid_difference</span> <span class="o">&lt;</span> <span class="n">tolerance</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Converged at iteration: </span><span class="si">{</span><span class="n">iteration</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Runtime: </span><span class="si">{</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_time</span><span class="si">}</span><span class="s1"> seconds&#39;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">cluster_labels</span><span class="p">,</span> <span class="n">centroids</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Did not converge in </span><span class="si">{</span><span class="n">max_iterations</span><span class="si">}</span><span class="s1"> iterations&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Runtime: </span><span class="si">{</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_time</span><span class="si">}</span><span class="s1"> seconds&#39;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">cluster_labels</span><span class="p">,</span> <span class="n">centroids</span>
<span class="c1"># quirk of numpy / Jupyter need to set seed again</span>
<span class="n">cluster_labels</span><span class="p">,</span> <span class="n">centroids</span> <span class="o">=</span> <span class="n">k_means</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Converged at iteration: 5
Runtime: 0.432811975479126 seconds
</pre></div>
</div>
</div>
</div>
<p><strong>Note</strong>: the start of the timing is after the random initialization, and first
‘cycle’ of our algorithm. This is technically not the correct way to time it but
due to this being in a Jupyter notebook and the way it is structured this way of
comparing our algorithms will produce a more equal result. When timing code we
should always encapsulate our whole computation block.</p>
<p>So we see an improvement from just switching to numpy’s argmin function. There
is a very nice tool (or category of tools) called profilers. These can be
utilized to make clearer which improvements to our code we should care most
about here is an <a class="reference external" href="https://ipython-books.github.io/42-profiling-your-code-easily-with-cprofile-and-ipython/">excellent source</a>
on the topic. Even before optimizing we can understand which parts of our code
will be taking the most of the run-time. It will be the longest Python loop,
i.e. the loop over all the samples. Nonetheless, let us do some profiling!</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">test_data</span> <span class="o">=</span> <span class="n">generate_simple_clustering_dataset</span><span class="p">(</span><span class="n">n_points</span><span class="o">=</span><span class="mi">10000</span><span class="p">,</span> <span class="n">plotting</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="o">%</span><span class="k">prun</span> -l 10 cluster_labels, centroids = k_means(test_data)
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Converged at iteration: 11
Runtime: 0.8653810024261475 seconds
</pre></div>
</div>
</div>
</div>
<p>Here we can see the reason for profiling. We now know for certain a lot can be
gained just by vectorizing our distance function. Ideally we wish to perform
most of our loops in numpy, i.e. C. To do this we need our array shapes to match
and clever reshaping will let us do so.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">np_get_distances_to_clusters</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">centroids</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Squared Euclidean distance between all data-points and every centroid. For</span>
<span class="sd"> the function to work properly it needs data and centroids to be numpy</span>
<span class="sd"> broadcastable. We sum along the dimension axis.</span>
<span class="sd"> Inputs:</span>
<span class="sd"> data (np.array): with dimensions (samples x 1 x dim)</span>
<span class="sd"> centroids (np.array): with dimensions (1 x n_clusters x dim)</span>
<span class="sd"> Returns:</span>
<span class="sd"> distances (np.array): with dimensions (samples x n_clusters)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">distances</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">((</span><span class="n">data</span> <span class="o">-</span> <span class="n">centroids</span><span class="p">))</span><span class="o">**</span><span class="mi">2</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="k">return</span> <span class="n">distances</span>
<span class="k">def</span> <span class="nf">np_assign_points_to_clusters</span><span class="p">(</span><span class="n">distances</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Assigning each data-point to a cluster given an array distances containing</span>
<span class="sd"> the squared Euclidean distance from every point to each centroid. We do</span>
<span class="sd"> np.argmin along the cluster axis to find the closest cluster. Returns a</span>
<span class="sd"> numpy array with corresponding labels.</span>
<span class="sd"> Inputs:</span>
<span class="sd"> distances (np.array): with dimensions (samples x n_clusters)</span>
<span class="sd"> Returns:</span>
<span class="sd"> cluster_labels (np.array): with dimensions (samples x None)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">cluster_labels</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">argmin</span><span class="p">(</span><span class="n">distances</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="k">return</span> <span class="n">cluster_labels</span>
<span class="k">def</span> <span class="nf">np_k_means</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">n_clusters</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">max_iterations</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">tolerance</span><span class="o">=</span><span class="mf">1e-8</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Numpythonic implementation of the k-means clusting algorithm.</span>
<span class="sd"> Inputs:</span>
<span class="sd"> data (np.array): with dimesions (samples x dim)</span>
<span class="sd"> n_clusters (int): hyperparameter which depends on dataset</span>
<span class="sd"> max_iterations (int): hyperparameter which depends on dataset</span>
<span class="sd"> tolerance (float): convergence measure</span>
<span class="sd"> progression_plot (bool): activation flag for plotting</span>
<span class="sd"> Returns:</span>
<span class="sd"> cluster_labels (np.array): with dimension (samples)</span>
<span class="sd"> centroid_list (list): list of centroids (np.array)</span>
<span class="sd"> with dimensions (n_clusters x dim)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">n_samples</span><span class="p">,</span> <span class="n">dimensions</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">shape</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2021</span><span class="p">)</span>
<span class="n">centroids</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="n">n_clusters</span><span class="p">,</span> <span class="n">replace</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="p">:]</span>
<span class="n">distances</span> <span class="o">=</span> <span class="n">np_get_distances_to_clusters</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">data</span><span class="p">,</span>
<span class="p">(</span><span class="n">n_samples</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">dimensions</span><span class="p">)),</span>
<span class="n">np</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">centroids</span><span class="p">,</span>
<span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">n_clusters</span><span class="p">,</span> <span class="n">dimensions</span><span class="p">)))</span>
<span class="n">cluster_labels</span> <span class="o">=</span> <span class="n">np_assign_points_to_clusters</span><span class="p">(</span><span class="n">distances</span><span class="p">)</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="k">for</span> <span class="n">iteration</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">max_iterations</span><span class="p">):</span>
<span class="n">prev_centroids</span> <span class="o">=</span> <span class="n">centroids</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">):</span>
<span class="n">points_in_cluster</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="n">cluster_labels</span> <span class="o">==</span> <span class="n">k</span><span class="p">]</span>
<span class="n">mean_vector</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">points_in_cluster</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">centroids</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">mean_vector</span>
<span class="n">distances</span> <span class="o">=</span> <span class="n">np_get_distances_to_clusters</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">data</span><span class="p">,</span>
<span class="p">(</span><span class="n">n_samples</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">dimensions</span><span class="p">)),</span>
<span class="n">np</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">centroids</span><span class="p">,</span>
<span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">n_clusters</span><span class="p">,</span> <span class="n">dimensions</span><span class="p">)))</span>
<span class="n">cluster_labels</span> <span class="o">=</span> <span class="n">np_assign_points_to_clusters</span><span class="p">(</span><span class="n">distances</span><span class="p">)</span>
<span class="n">centroid_difference</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">centroids</span> <span class="o">-</span> <span class="n">prev_centroids</span><span class="p">))</span>
<span class="k">if</span> <span class="n">centroid_difference</span> <span class="o">&lt;</span> <span class="n">tolerance</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Converged at iteration: </span><span class="si">{</span><span class="n">iteration</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Runtime: </span><span class="si">{</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_time</span><span class="si">}</span><span class="s1"> seconds&#39;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">cluster_labels</span><span class="p">,</span> <span class="n">centroids</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Did not converge in </span><span class="si">{</span><span class="n">max_iterations</span><span class="si">}</span><span class="s1"> iterations&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Runtime: </span><span class="si">{</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_time</span><span class="si">}</span><span class="s1"> seconds&#39;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">cluster_labels</span><span class="p">,</span> <span class="n">centroids</span>
</pre></div>
</div>
</div>
</div>
<p>When working towards becoming a data scientist using Python this last step is
arguably one of the most important. Thinking of ways to avoid explicitly looping
by adding dimensions to our arrays in such a way that they become broadcastable
using numpy (also tensorflow and many others). Let us take a look at our the
fruits of our labor.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">cluster_labels</span><span class="p">,</span> <span class="n">centroids</span> <span class="o">=</span> <span class="n">np_k_means</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Converged at iteration: 5
Runtime: 0.0036406517028808594 seconds
</pre></div>
</div>
</div>
</div>
</div>
</div>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
kernelName: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</div>
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter8.html" title="previous page"><span class="section-number">11. </span>Basic ideas of the Principal Component Analysis (PCA)</a>
<a class='right-next' id="next-link" href="chapter9.html" title="next page"><span class="section-number">13. </span>Neural networks</a>
</div>
</div>
</div>
<footer class="footer mt-5 mt-md-0">
<div class="container">
<p>
By Morten Hjorth-Jensen<br/>
&copy; Copyright 2021.<br/>
</p>
</div>
</footer>
</main>
</div>
</div>
<script src="_static/js/index.1c5a1a01449ed65a7b51.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink