lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-12-08 06:31:52 +01:00
parent 5bd3b65a7a
commit 9fdff62ca4
15 changed files with 6571 additions and 2626 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/BookChapters/chapter1.dlog
+7
View File
@@ -43,3 +43,10 @@ Translating doconce text in chapter1.do.txt to ipynb
Failed to remove ans_at_end environment
Failed to remove sol_at_end environment
output in chapter1.ipynb
Translating doconce text in chapter1.do.txt to ipynb
*** replacing \bm{...} by \boldsymbol{...} (\bm is not supported by MathJax)
*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments.
*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments.
output in chapter1.ipynb
doc/BookChapters/chapter1.ipynb
+4320
View File
File diff suppressed because it is too large Load Diff
doc/BookChapters/chapter10.do.txt
+1 -4
View File
@@ -1009,10 +1009,7 @@ Program Interfaces (APIs), and see how we use them to build our graph.
Then we will build (effectively) the same graph in Keras, to see just
how simple solving a machine learning problem can be.
To install tensorflow on Unix/Linux systems, use pip as
!bc pycod
pip3 install tensorflow
!ec
To install tensorflow on Unix/Linux systems, use pip as _pip3 install tensorflow_
and/or if you use _anaconda_, just write (or install from the graphical user interface)
(current release of CPU-only TensorFlow)
!bc pycod
doc/BookChapters/chapter10.ipynb
-2047
View File
File diff suppressed because it is too large Load Diff
doc/BookChapters/chapter11.ipynb
+528 -176
View File
File diff suppressed because it is too large Load Diff
doc/BookChapters/clustering.do.txt
+416
View File
@@ -0,0 +1,416 @@
======= Clustering and Unsupervised Learning =======
In general terms cluster analysis, or clustering, is the task of grouping a
data-set into different distinct categories based on some measure of equality of
the data. This measure is often referred to as a _metric_ or _similarity
measure_ in the literature (note: sometimes we deal with a _dissimilarity
measure_ instead). Usually, these metrics are formulated as some kind of
distance function between points in a high-dimensional space.
The simplest, and also the most
common is the _Euclidean distance_.
The simplest of all clustering algorithms is the _k-means algorithm_
, sometimes also referred to as *Lloyds algorithm*. It is the simplest and also
the most common. From its simplicity it obtains both strengths and weaknesses.
These will be discussed in more detail later. The $k$-means algorithm is a
_centroid based_ clustering algorithm.
Assume, we are given $n$ data points and we wish to split the data into $K < n$
different categories, or clusters. We label each cluster by an integer
!bt
\[ k\in\{1, \cdots, K \}.
\]
!et
In the basic k-means algorithm each point is assigned to only
one cluster $k$, and these assignments are *non-injective* i.e. many-to-one. We
can think of these mappings as an encoder $k = C(i)$, which assigns the $i$-th
data-point $\bf x_i$ to the $k$-th cluster.
$k$-means algorithm in words:
o We start with guesses / random initializations of our $k$ cluster centers/centroids
o For each centroid the points that are most similar are identified
o Then we move / replace each centroid with a coordinate average of all the points that were assigned to that centroid.
o Iterate 2-3 until the centroids no longer move (to some tolerance)
We assume we have $n$ data-points
!bt
\begin{equation}\label{eq:kmeanspoints}
\bm{x_i} = \{x_{i, 1}, \cdots, x_{i, p}\}\in\mathbb{R}^p.
\end{equation}
!et
which we wish to group into $K < n$ clusters. For our dissimilarity measure we
use the *squared Euclidean distance*
!bt
\begin{equation}\label{eq:squaredeuclidean}
d(\bm{x_i}, \bm{x_i'}) = \sum_{j=1}^p(x_{ij} - x_{i'j})^2
= ||\bm{x_i} - \bm{x_{i'}}||^2
\end{equation}
!et
We define the so called *within-cluster point scatter* which gives us a
measure of how close each data point assigned to the same cluster tends to be to
the all the others.
!bt
\begin{equation}\label{eq:withincluster}
W(C) = \frac{1}{2}\sum_{k=1}^K\sum_{C(i)=k}
\sum_{C(i')=k}d(\bm{x_i}, \bm{x_{i'}}) =
\sum_{k=1}^KN_k\sum_{C(i)=k}||\bm{x_i} - \bm{\overline{x_k}}||^2
\end{equation}
!et
where $\bm{\overline{x_k}}$ is the mean vector associated with the $k$-th
cluster, and $N_k = \sum_{i=1}^nI(C(i) = k)$, where the $I()$ notation is
similar to the Kronecker delta (*Commonly used in statistics, it just means that
when $i = k$ we have the encoder $C(i)$*). In other words, the within-cluster
scatter measures the compactness of each cluster with respect to the data points
assigned to each cluster. This is the quantity that the $k$-means algorithm aims
to minimize. We refer to this quantity $W(C)$ as the within cluster scatter
because of its relation to the *total scatter*.
We have
!bt
\begin{equation}\label{eq:totalscatter}
T = W(C) + B(C) = \frac{1}{2}\sum_{i=1}^n
\sum_{i'=1}^nd(\bm{x_i}, \bm{x_{i'}})
= \frac{1}{2}\sum_{k=1}^K\sum_{C(i)=k}
\Big(\sum_{C(i') = k}d(\bm{x_i}, \bm{x_{i'}})
+ \sum_{C(i')\neq k}d(\bm{x_i}, \bm{x_{i'}})\Big).
\end{equation}
!et
This is a quantity that is conserved throughout the $k$-means algorithm. It can
be thought of as the total amount of information in the data, and it is composed
of the aforementioned within-cluster scatter and the *between-cluster scatter*
$B(C)$. In methods such as principle component analysis the total scatter is not
conserved.
Given a cluster mean $\bm{m_k}$ we define the _total cluster variance_
!bt
\begin{equation}\label{eq:totalclustervariance}
\min_{C, \{\bm{m_k}\}_1^K}\sum_{k=1}^KN_k\sum||\bm{x_i} - \bm{m_k}||^2
\end{equation}
!et
Now we have all the pieces necessary to formally revisit the $k$-means algorithm.
The $k$-means clustering algorithm goes as follows
o For a given cluster assignment $C$, and $k$ cluster means $\left\{m_1, \cdots, m_k\right\}$. We minimize the total cluster variance with respect to the cluster means $\{m_k\}$ yielding the means of the currently assigned clusters.
o Given a current set of $k$ means $\{m_k\}$ the total cluster variance is minimized by assigning each observation to the closest (current) cluster mean. That is $$C(i) = \underset{1\leq k\leq K}{\mathrm{argmin}} ||\bm{x_i} - \bm{m_k}||^2$$
o Steps 1 and 2 are repeated until the assignments do not change.
===== Codes and Approaches =====
o Before we start we specify a number $k$ which is the number of clusters we want to try to separate our data into.
o We initially choose $k$ random data points in our data as our initial centroids, *or means* (this is where the name comes from).
o Assign each data point to their closest centroid, based on the squared Euclidean distance.
o For each of the $k$ cluster we update the centroid by calculating new mean values for all the data points in the cluster.
o Iteratively minimize the within cluster scatter by performing steps (3, 4) until the new assignments stop changing (can be to some tolerance) or until a maximum number of iterations have passed.
Let us now program the most basic version of the algorithm using nothing but
Python with numpy arrays. This code is kept intentionally simple to gradually
progress our understanding. There is no vectorization of any kind, and even most
helper functions are not utilized.
We need first a dataset to do our cluster analysis on. In our case
this is a plain *vanilla* data set using random numbers using a
Gaussian distribution.
!bc pycod
import time
import numpy as np
import tensorflow as tf
from matplotlib import image
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from IPython.display import display
np.random.seed(2021)
!ec
Next we define functions, for ease of use later, to generate Gaussians and to
set up our toy data set.
!bc pycod
def gaussian_points(dim=2, n_points=1000, mean_vector=np.array([0, 0]),
sample_variance=1):
"""
Very simple custom function to generate gaussian distributed point clusters
with variable dimension, number of points, means in each direction
(must match dim) and sample variance.
Inputs:
dim (int)
n_points (int)
mean_vector (np.array) (where index 0 is x, index 1 is y etc.)
sample_variance (float)
Returns:
data (np.array): with dimensions (dim x n_points)
"""
mean_matrix = np.zeros(dim) + mean_vector
covariance_matrix = np.eye(dim) * sample_variance
data = np.random.multivariate_normal(mean_matrix, covariance_matrix,
n_points)
return data
def generate_simple_clustering_dataset(dim=2, n_points=1000, plotting=True,
return_data=True):
"""
Toy model to illustrate k-means clustering
"""
data1 = gaussian_points(mean_vector=np.array([5, 5]))
data2 = gaussian_points()
data3 = gaussian_points(mean_vector=np.array([1, 4.5]))
data4 = gaussian_points(mean_vector=np.array([5, 1]))
data = np.concatenate((data1, data2, data3, data4), axis=0)
if plotting:
fig, ax = plt.subplots()
ax.scatter(data[:, 0], data[:, 1], alpha=0.2)
ax.set_title('Toy Model Dataset')
plt.show()
if return_data:
return data
data = generate_simple_clustering_dataset()
!ec
With the above dataset we start
implementing the $k$-means algorithm.
!bc pycod
n_samples, dimensions = data.shape
n_clusters = 4
# we randomly initialize our centroids
np.random.seed(2021)
centroids = data[np.random.choice(n_samples, n_clusters, replace=False), :]
distances = np.zeros((n_samples, n_clusters))
# first we need to calculate the distance to each centroid from our data
for k in range(n_clusters):
for n in range(n_samples):
dist = 0
for d in range(dimensions):
dist += np.abs(data[n, d] - centroids[k, d])**2
distances[n, k] = dist
# we initialize an array to keep track of to which cluster each point belongs
# the way we set it up here the index tracks which point and the value which
# cluster the point belongs to
cluster_labels = np.zeros(n_samples, dtype='int')
# next we loop through our samples and for every point assign it to the cluster
# to which it has the smallest distance to
for n in range(n_samples):
# tracking variables (all of this is basically just an argmin)
smallest = 1e10
smallest_row_index = 1e10
for k in range(n_clusters):
if distances[n, k] < smallest:
smallest = distances[n, k]
smallest_row_index = k
cluster_labels[n] = smallest_row_index
!ec
!bc pycod
fig = plt.figure()
ax = fig.add_subplot()
unique_cluster_labels = np.unique(cluster_labels)
for i in unique_cluster_labels:
ax.scatter(data[cluster_labels == i, 0],
data[cluster_labels == i, 1],
label = i,
alpha = 0.2)
ax.scatter(centroids[:, 0], centroids[:, 1], c='black')
ax.set_title("First Grouping of Points to Centroids")
plt.show()
!ec
So what do we have so far? We have 'picked' $k$ centroids at random from our
data points. There are other ways of more intelligently choosing their
initializations, however for our purposes randomly is fine. Then we have
initialized an array 'distances' which holds the information of the distance,
*or dissimilarity*, of every point to of our centroids. Finally, we have
initialized an array 'cluster_labels' which according to our distances array
holds the information of to which centroid every point is assigned. This was the
first pass of our algorithm. Essentially, all we need to do now is repeat the
distance and assignment steps above until we have reached a desired convergence
or a maximum amount of iterations.
!bc pycod
max_iterations = 100
tolerance = 1e-8
for iteration in range(max_iterations):
prev_centroids = centroids.copy()
for k in range(n_clusters):
# this array will be used to update our centroid positions
vector_mean = np.zeros(dimensions)
mean_divisor = 0
for n in range(n_samples):
if cluster_labels[n] == k:
vector_mean += data[n, :]
mean_divisor += 1
# update according to the k means
centroids[k, :] = vector_mean / mean_divisor
# we find the dissimilarity
for k in range(n_clusters):
for n in range(n_samples):
dist = 0
for d in range(dimensions):
dist += np.abs(data[n, d] - centroids[k, d])**2
distances[n, k] = dist
# assign each point
for n in range(n_samples):
smallest = 1e10
smallest_row_index = 1e10
for k in range(n_clusters):
if distances[n, k] < smallest:
smallest = distances[n, k]
smallest_row_index = k
cluster_labels[n] = smallest_row_index
# convergence criteria
centroid_difference = np.sum(np.abs(centroids - prev_centroids))
if centroid_difference < tolerance:
print(f'Converged at iteration {iteration}')
break
elif iteration == max_iterations:
print(f'Did not converge in {max_iterations} iterations')
!ec
We now have a simple , un-optimized $k$-means
clustering implementation. Lets plot the final result
!bc pycod
fig = plt.figure()
ax = fig.add_subplot()
unique_cluster_labels = np.unique(cluster_labels)
for i in unique_cluster_labels:
ax.scatter(data[cluster_labels == i, 0],
data[cluster_labels == i, 1],
label = i,
alpha = 0.2)
ax.scatter(centroids[:, 0], centroids[:, 1], c='black')
ax.set_title("Final Result of K-means Clustering")
plt.show()
!ec
!bc pycod
def naive_kmeans(data, n_clusters=4, max_iterations=100, tolerance=1e-8):
start_time = time.time()
n_samples, dimensions = data.shape
n_clusters = 4
#np.random.seed(2021)
centroids = data[np.random.choice(n_samples, n_clusters, replace=False), :]
distances = np.zeros((n_samples, n_clusters))
for k in range(n_clusters):
for n in range(n_samples):
dist = 0
for d in range(dimensions):
dist += np.abs(data[n, d] - centroids[k, d])**2
distances[n, k] = dist
cluster_labels = np.zeros(n_samples, dtype='int')
for n in range(n_samples):
smallest = 1e10
smallest_row_index = 1e10
for k in range(n_clusters):
if distances[n, k] < smallest:
smallest = distances[n, k]
smallest_row_index = k
cluster_labels[n] = smallest_row_index
for iteration in range(max_iterations):
prev_centroids = centroids.copy()
for k in range(n_clusters):
vector_mean = np.zeros(dimensions)
mean_divisor = 0
for n in range(n_samples):
if cluster_labels[n] == k:
vector_mean += data[n, :]
mean_divisor += 1
centroids[k, :] = vector_mean / mean_divisor
for k in range(n_clusters):
for n in range(n_samples):
dist = 0
for d in range(dimensions):
dist += np.abs(data[n, d] - centroids[k, d])**2
distances[n, k] = dist
for n in range(n_samples):
smallest = 1e10
smallest_row_index = 1e10
for k in range(n_clusters):
if distances[n, k] < smallest:
smallest = distances[n, k]
smallest_row_index = k
cluster_labels[n] = smallest_row_index
centroid_difference = np.sum(np.abs(centroids - prev_centroids))
if centroid_difference < tolerance:
print(f'Converged at iteration {iteration}')
print(f'Runtime: {time.time() - start_time} seconds')
return cluster_labels, centroids
print(f'Did not converge in {max_iterations} iterations')
print(f'Runtime: {time.time() - start_time} seconds')
return cluster_labels, centroids
!ec
doc/LectureNotes/chapter10.ipynb
+417 -166
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/clustering.ipynb
+649
View File
@@ -0,0 +1,649 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "c3edcae5",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html clustering.do.txt -->"
]
},
{
"cell_type": "markdown",
"id": "4102577e",
"metadata": {
"editable": true
},
"source": [
"# Clustering and Unsupervised Learning\n",
"\n",
"In general terms cluster analysis, or clustering, is the task of grouping a\n",
"data-set into different distinct categories based on some measure of equality of\n",
"the data. This measure is often referred to as a **metric** or **similarity\n",
"measure** in the literature (note: sometimes we deal with a **dissimilarity\n",
"measure** instead). Usually, these metrics are formulated as some kind of\n",
"distance function between points in a high-dimensional space.\n",
"\n",
"The simplest, and also the most\n",
"common is the **Euclidean distance**.\n",
"\n",
"The simplest of all clustering algorithms is the **k-means algorithm**\n",
", sometimes also referred to as *Lloyds algorithm*. It is the simplest and also\n",
"the most common. From its simplicity it obtains both strengths and weaknesses.\n",
"These will be discussed in more detail later. The $k$-means algorithm is a\n",
"**centroid based** clustering algorithm.\n",
"\n",
"Assume, we are given $n$ data points and we wish to split the data into $K < n$\n",
"different categories, or clusters. We label each cluster by an integer"
]
},
{
"cell_type": "markdown",
"id": "0deb3255",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"k\\in\\{1, \\cdots, K \\}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "cfd8fe00",
"metadata": {
"editable": true
},
"source": [
"In the basic k-means algorithm each point is assigned to only\n",
"one cluster $k$, and these assignments are *non-injective* i.e. many-to-one. We\n",
"can think of these mappings as an encoder $k = C(i)$, which assigns the $i$-th\n",
"data-point $\\bf x_i$ to the $k$-th cluster.\n",
"\n",
"$k$-means algorithm in words:\n",
"1. We start with guesses / random initializations of our $k$ cluster centers/centroids\n",
"\n",
"2. For each centroid the points that are most similar are identified\n",
"\n",
"3. Then we move / replace each centroid with a coordinate average of all the points that were assigned to that centroid.\n",
"\n",
"4. Iterate 2-3 until the centroids no longer move (to some tolerance)\n",
"\n",
"We assume we have $n$ data-points"
]
},
{
"cell_type": "markdown",
"id": "a29b7459",
"metadata": {
"editable": true
},
"source": [
"<!-- Equation labels as ordinary links -->\n",
"<div id=\"eq:kmeanspoints\"></div>\n",
"\n",
"$$\n",
"\\begin{equation}\\label{eq:kmeanspoints} \\tag{1}\n",
" \\boldsymbol{x_i} = \\{x_{i, 1}, \\cdots, x_{i, p}\\}\\in\\mathbb{R}^p.\n",
"\\end{equation}\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "98f9e37b",
"metadata": {
"editable": true
},
"source": [
"which we wish to group into $K < n$ clusters. For our dissimilarity measure we\n",
"use the *squared Euclidean distance*"
]
},
{
"cell_type": "markdown",
"id": "d3c32572",
"metadata": {
"editable": true
},
"source": [
"<!-- Equation labels as ordinary links -->\n",
"<div id=\"eq:squaredeuclidean\"></div>\n",
"\n",
"$$\n",
"\\begin{equation}\\label{eq:squaredeuclidean} \\tag{2}\n",
" d(\\boldsymbol{x_i}, \\boldsymbol{x_i'}) = \\sum_{j=1}^p(x_{ij} - x_{i'j})^2\n",
" = ||\\boldsymbol{x_i} - \\boldsymbol{x_{i'}}||^2\n",
"\\end{equation}\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "29d24648",
"metadata": {
"editable": true
},
"source": [
"We define the so called *within-cluster point scatter* which gives us a\n",
"measure of how close each data point assigned to the same cluster tends to be to\n",
"the all the others."
]
},
{
"cell_type": "markdown",
"id": "fce5c797",
"metadata": {
"editable": true
},
"source": [
"<!-- Equation labels as ordinary links -->\n",
"<div id=\"eq:withincluster\"></div>\n",
"\n",
"$$\n",
"\\begin{equation}\\label{eq:withincluster} \\tag{3}\n",
" W(C) = \\frac{1}{2}\\sum_{k=1}^K\\sum_{C(i)=k}\n",
" \\sum_{C(i')=k}d(\\boldsymbol{x_i}, \\boldsymbol{x_{i'}}) =\n",
" \\sum_{k=1}^KN_k\\sum_{C(i)=k}||\\boldsymbol{x_i} - \\boldsymbol{\\overline{x_k}}||^2\n",
"\\end{equation}\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "674a26b7",
"metadata": {
"editable": true
},
"source": [
"where $\\boldsymbol{\\overline{x_k}}$ is the mean vector associated with the $k$-th\n",
"cluster, and $N_k = \\sum_{i=1}^nI(C(i) = k)$, where the $I()$ notation is\n",
"similar to the Kronecker delta (*Commonly used in statistics, it just means that\n",
"when $i = k$ we have the encoder $C(i)$*). In other words, the within-cluster\n",
"scatter measures the compactness of each cluster with respect to the data points\n",
"assigned to each cluster. This is the quantity that the $k$-means algorithm aims\n",
"to minimize. We refer to this quantity $W(C)$ as the within cluster scatter\n",
"because of its relation to the *total scatter*.\n",
"\n",
"We have"
]
},
{
"cell_type": "markdown",
"id": "f200e7ff",
"metadata": {
"editable": true
},
"source": [
"<!-- Equation labels as ordinary links -->\n",
"<div id=\"eq:totalscatter\"></div>\n",
"\n",
"$$\n",
"\\begin{equation}\\label{eq:totalscatter} \\tag{4}\n",
" T = W(C) + B(C) = \\frac{1}{2}\\sum_{i=1}^n\n",
" \\sum_{i'=1}^nd(\\boldsymbol{x_i}, \\boldsymbol{x_{i'}})\n",
" = \\frac{1}{2}\\sum_{k=1}^K\\sum_{C(i)=k}\n",
" \\Big(\\sum_{C(i') = k}d(\\boldsymbol{x_i}, \\boldsymbol{x_{i'}})\n",
" + \\sum_{C(i')\\neq k}d(\\boldsymbol{x_i}, \\boldsymbol{x_{i'}})\\Big).\n",
"\\end{equation}\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "5471a94d",
"metadata": {
"editable": true
},
"source": [
"This is a quantity that is conserved throughout the $k$-means algorithm. It can\n",
"be thought of as the total amount of information in the data, and it is composed\n",
"of the aforementioned within-cluster scatter and the *between-cluster scatter*\n",
"$B(C)$. In methods such as principle component analysis the total scatter is not\n",
"conserved.\n",
"\n",
"Given a cluster mean $\\boldsymbol{m_k}$ we define the **total cluster variance**"
]
},
{
"cell_type": "markdown",
"id": "299a99ce",
"metadata": {
"editable": true
},
"source": [
"<!-- Equation labels as ordinary links -->\n",
"<div id=\"eq:totalclustervariance\"></div>\n",
"\n",
"$$\n",
"\\begin{equation}\\label{eq:totalclustervariance} \\tag{5}\n",
" \\min_{C, \\{\\boldsymbol{m_k}\\}_1^K}\\sum_{k=1}^KN_k\\sum||\\boldsymbol{x_i} - \\boldsymbol{m_k}||^2\n",
"\\end{equation}\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "bdfa54ee",
"metadata": {
"editable": true
},
"source": [
"Now we have all the pieces necessary to formally revisit the $k$-means algorithm.\n",
"\n",
"The $k$-means clustering algorithm goes as follows \n",
"\n",
"1. For a given cluster assignment $C$, and $k$ cluster means $\\left\\{m_1, \\cdots, m_k\\right\\}$. We minimize the total cluster variance with respect to the cluster means $\\{m_k\\}$ yielding the means of the currently assigned clusters.\n",
"\n",
"2. Given a current set of $k$ means $\\{m_k\\}$ the total cluster variance is minimized by assigning each observation to the closest (current) cluster mean. That is $$C(i) = \\underset{1\\leq k\\leq K}{\\mathrm{argmin}} ||\\boldsymbol{x_i} - \\boldsymbol{m_k}||^2$$\n",
"\n",
"3. Steps 1 and 2 are repeated until the assignments do not change."
]
},
{
"cell_type": "markdown",
"id": "f5def86c",
"metadata": {
"editable": true
},
"source": [
"## Codes and Approaches\n",
"\n",
"1. Before we start we specify a number $k$ which is the number of clusters we want to try to separate our data into.\n",
"\n",
"2. We initially choose $k$ random data points in our data as our initial centroids, *or means* (this is where the name comes from).\n",
"\n",
"3. Assign each data point to their closest centroid, based on the squared Euclidean distance.\n",
"\n",
"4. For each of the $k$ cluster we update the centroid by calculating new mean values for all the data points in the cluster.\n",
"\n",
"5. Iteratively minimize the within cluster scatter by performing steps (3, 4) until the new assignments stop changing (can be to some tolerance) or until a maximum number of iterations have passed.\n",
"\n",
"Let us now program the most basic version of the algorithm using nothing but\n",
"Python with numpy arrays. This code is kept intentionally simple to gradually\n",
"progress our understanding. There is no vectorization of any kind, and even most\n",
"helper functions are not utilized.\n",
"\n",
"We need first a dataset to do our cluster analysis on. In our case\n",
"this is a plain *vanilla* data set using random numbers using a\n",
"Gaussian distribution."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "b0260188",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"import time\n",
"import numpy as np\n",
"import tensorflow as tf\n",
"from matplotlib import image\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.cluster import KMeans\n",
"from IPython.display import display\n",
"\n",
"np.random.seed(2021)"
]
},
{
"cell_type": "markdown",
"id": "fe680e35",
"metadata": {
"editable": true
},
"source": [
"Next we define functions, for ease of use later, to generate Gaussians and to\n",
"set up our toy data set."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9db2bbce",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"def gaussian_points(dim=2, n_points=1000, mean_vector=np.array([0, 0]),\n",
" sample_variance=1):\n",
" \"\"\"\n",
" Very simple custom function to generate gaussian distributed point clusters\n",
" with variable dimension, number of points, means in each direction\n",
" (must match dim) and sample variance.\n",
"\n",
" Inputs:\n",
" dim (int)\n",
" n_points (int)\n",
" mean_vector (np.array) (where index 0 is x, index 1 is y etc.)\n",
" sample_variance (float)\n",
"\n",
" Returns:\n",
" data (np.array): with dimensions (dim x n_points)\n",
" \"\"\"\n",
"\n",
" mean_matrix = np.zeros(dim) + mean_vector\n",
" covariance_matrix = np.eye(dim) * sample_variance\n",
" data = np.random.multivariate_normal(mean_matrix, covariance_matrix,\n",
" n_points)\n",
" return data\n",
"\n",
"\n",
"\n",
"def generate_simple_clustering_dataset(dim=2, n_points=1000, plotting=True,\n",
" return_data=True):\n",
" \"\"\"\n",
" Toy model to illustrate k-means clustering\n",
" \"\"\"\n",
"\n",
" data1 = gaussian_points(mean_vector=np.array([5, 5]))\n",
" data2 = gaussian_points()\n",
" data3 = gaussian_points(mean_vector=np.array([1, 4.5]))\n",
" data4 = gaussian_points(mean_vector=np.array([5, 1]))\n",
" data = np.concatenate((data1, data2, data3, data4), axis=0)\n",
"\n",
" if plotting:\n",
" fig, ax = plt.subplots()\n",
" ax.scatter(data[:, 0], data[:, 1], alpha=0.2)\n",
" ax.set_title('Toy Model Dataset')\n",
" plt.show()\n",
"\n",
"\n",
" if return_data:\n",
" return data\n",
"\n",
"\n",
"data = generate_simple_clustering_dataset()"
]
},
{
"cell_type": "markdown",
"id": "c0bb8c76",
"metadata": {
"editable": true
},
"source": [
"With the above dataset we start\n",
"implementing the $k$-means algorithm."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "29a75065",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"\n",
"n_samples, dimensions = data.shape\n",
"n_clusters = 4\n",
"\n",
"# we randomly initialize our centroids\n",
"np.random.seed(2021)\n",
"centroids = data[np.random.choice(n_samples, n_clusters, replace=False), :]\n",
"distances = np.zeros((n_samples, n_clusters))\n",
"\n",
"# first we need to calculate the distance to each centroid from our data\n",
"for k in range(n_clusters):\n",
" for n in range(n_samples):\n",
" dist = 0\n",
" for d in range(dimensions):\n",
" dist += np.abs(data[n, d] - centroids[k, d])**2\n",
" distances[n, k] = dist\n",
"\n",
"# we initialize an array to keep track of to which cluster each point belongs\n",
"# the way we set it up here the index tracks which point and the value which\n",
"# cluster the point belongs to\n",
"cluster_labels = np.zeros(n_samples, dtype='int')\n",
"\n",
"# next we loop through our samples and for every point assign it to the cluster\n",
"# to which it has the smallest distance to\n",
"for n in range(n_samples):\n",
" # tracking variables (all of this is basically just an argmin)\n",
" smallest = 1e10\n",
" smallest_row_index = 1e10\n",
" for k in range(n_clusters):\n",
" if distances[n, k] < smallest:\n",
" smallest = distances[n, k]\n",
" smallest_row_index = k\n",
"\n",
" cluster_labels[n] = smallest_row_index"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "9fae7fc9",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"fig = plt.figure()\n",
"ax = fig.add_subplot()\n",
"unique_cluster_labels = np.unique(cluster_labels)\n",
"for i in unique_cluster_labels:\n",
" ax.scatter(data[cluster_labels == i, 0],\n",
" data[cluster_labels == i, 1],\n",
" label = i,\n",
" alpha = 0.2)\n",
" ax.scatter(centroids[:, 0], centroids[:, 1], c='black')\n",
"\n",
"ax.set_title(\"First Grouping of Points to Centroids\")\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "90d2a873",
"metadata": {
"editable": true
},
"source": [
"So what do we have so far? We have 'picked' $k$ centroids at random from our\n",
"data points. There are other ways of more intelligently choosing their\n",
"initializations, however for our purposes randomly is fine. Then we have\n",
"initialized an array 'distances' which holds the information of the distance,\n",
"*or dissimilarity*, of every point to of our centroids. Finally, we have\n",
"initialized an array 'cluster_labels' which according to our distances array\n",
"holds the information of to which centroid every point is assigned. This was the\n",
"first pass of our algorithm. Essentially, all we need to do now is repeat the\n",
"distance and assignment steps above until we have reached a desired convergence\n",
"or a maximum amount of iterations."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "378c29fc",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"\n",
"max_iterations = 100\n",
"tolerance = 1e-8\n",
"\n",
"for iteration in range(max_iterations):\n",
" prev_centroids = centroids.copy()\n",
" for k in range(n_clusters):\n",
" # this array will be used to update our centroid positions\n",
" vector_mean = np.zeros(dimensions)\n",
" mean_divisor = 0\n",
" for n in range(n_samples):\n",
" if cluster_labels[n] == k:\n",
" vector_mean += data[n, :]\n",
" mean_divisor += 1\n",
"\n",
" # update according to the k means\n",
" centroids[k, :] = vector_mean / mean_divisor\n",
"\n",
" # we find the dissimilarity\n",
" for k in range(n_clusters):\n",
" for n in range(n_samples):\n",
" dist = 0\n",
" for d in range(dimensions):\n",
" dist += np.abs(data[n, d] - centroids[k, d])**2\n",
" distances[n, k] = dist\n",
"\n",
" # assign each point\n",
" for n in range(n_samples):\n",
" smallest = 1e10\n",
" smallest_row_index = 1e10\n",
" for k in range(n_clusters):\n",
" if distances[n, k] < smallest:\n",
" smallest = distances[n, k]\n",
" smallest_row_index = k\n",
"\n",
" cluster_labels[n] = smallest_row_index\n",
"\n",
" # convergence criteria\n",
" centroid_difference = np.sum(np.abs(centroids - prev_centroids))\n",
" if centroid_difference < tolerance:\n",
" print(f'Converged at iteration {iteration}')\n",
" break\n",
"\n",
" elif iteration == max_iterations:\n",
" print(f'Did not converge in {max_iterations} iterations')"
]
},
{
"cell_type": "markdown",
"id": "545a6742",
"metadata": {
"editable": true
},
"source": [
"We now have a simple , un-optimized $k$-means\n",
"clustering implementation. Lets plot the final result"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "d9d3973b",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"fig = plt.figure()\n",
"ax = fig.add_subplot()\n",
"unique_cluster_labels = np.unique(cluster_labels)\n",
"for i in unique_cluster_labels:\n",
" ax.scatter(data[cluster_labels == i, 0],\n",
" data[cluster_labels == i, 1],\n",
" label = i,\n",
" alpha = 0.2)\n",
" ax.scatter(centroids[:, 0], centroids[:, 1], c='black')\n",
"\n",
"ax.set_title(\"Final Result of K-means Clustering\")\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "ee6a145f",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"def naive_kmeans(data, n_clusters=4, max_iterations=100, tolerance=1e-8):\n",
" start_time = time.time()\n",
"\n",
" n_samples, dimensions = data.shape\n",
" n_clusters = 4\n",
" #np.random.seed(2021)\n",
" centroids = data[np.random.choice(n_samples, n_clusters, replace=False), :]\n",
" distances = np.zeros((n_samples, n_clusters))\n",
"\n",
" for k in range(n_clusters):\n",
" for n in range(n_samples):\n",
" dist = 0\n",
" for d in range(dimensions):\n",
" dist += np.abs(data[n, d] - centroids[k, d])**2\n",
" distances[n, k] = dist\n",
"\n",
" cluster_labels = np.zeros(n_samples, dtype='int')\n",
"\n",
" for n in range(n_samples):\n",
" smallest = 1e10\n",
" smallest_row_index = 1e10\n",
" for k in range(n_clusters):\n",
" if distances[n, k] < smallest:\n",
" smallest = distances[n, k]\n",
" smallest_row_index = k\n",
"\n",
" cluster_labels[n] = smallest_row_index\n",
"\n",
" for iteration in range(max_iterations):\n",
" prev_centroids = centroids.copy()\n",
" for k in range(n_clusters):\n",
" vector_mean = np.zeros(dimensions)\n",
" mean_divisor = 0\n",
" for n in range(n_samples):\n",
" if cluster_labels[n] == k:\n",
" vector_mean += data[n, :]\n",
" mean_divisor += 1\n",
"\n",
" centroids[k, :] = vector_mean / mean_divisor\n",
"\n",
" for k in range(n_clusters):\n",
" for n in range(n_samples):\n",
" dist = 0\n",
" for d in range(dimensions):\n",
" dist += np.abs(data[n, d] - centroids[k, d])**2\n",
" distances[n, k] = dist\n",
"\n",
" for n in range(n_samples):\n",
" smallest = 1e10\n",
" smallest_row_index = 1e10\n",
" for k in range(n_clusters):\n",
" if distances[n, k] < smallest:\n",
" smallest = distances[n, k]\n",
" smallest_row_index = k\n",
"\n",
" cluster_labels[n] = smallest_row_index\n",
"\n",
" centroid_difference = np.sum(np.abs(centroids - prev_centroids))\n",
" if centroid_difference < tolerance:\n",
" print(f'Converged at iteration {iteration}')\n",
" print(f'Runtime: {time.time() - start_time} seconds')\n",
"\n",
" return cluster_labels, centroids\n",
"\n",
" print(f'Did not converge in {max_iterations} iterations')\n",
" print(f'Runtime: {time.time() - start_time} seconds')\n",
"\n",
" return cluster_labels, centroids"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
doc/pub/week47/html/._week47-bs000.html
+1 -1
View File
@@ -369,7 +369,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Nov 26, 2021</h4>
<h4>Dec 8, 2021</h4>
</center> <!-- date -->
<br>
doc/pub/week47/html/week47-bs.html
+1 -1
View File
@@ -369,7 +369,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Nov 26, 2021</h4>
<h4>Dec 8, 2021</h4>
</center> <!-- date -->
<br>
doc/pub/week47/html/week47-reveal.html
+1 -1
View File
@@ -184,7 +184,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Nov 26, 2021</h4>
<h4>Dec 8, 2021</h4>
</center> <!-- date -->
<br>
doc/pub/week47/html/week47-solarized.html
+1 -1
View File
@@ -293,7 +293,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Nov 26, 2021</h4>
<h4>Dec 8, 2021</h4>
</center> <!-- date -->
<br>
doc/pub/week47/html/week47.html
+1 -1
View File
@@ -370,7 +370,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Nov 26, 2021</h4>
<h4>Dec 8, 2021</h4>
</center> <!-- date -->
<br>
doc/pub/week47/ipynb/ipynb-week47-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week47/ipynb/week47.ipynb
+228 -228
View File
File diff suppressed because it is too large Load Diff