Finding structure without labels

Sometimes there are no labels: no column saying which customers churned, no file saying which documents matter. This section covers what can still be learned from the shape of the data alone, grouping similar things together, squeezing many dimensions down to a few you can plot, and spotting the examples unlike everything else. It also covers the honest limit, which is that an algorithm will always return groups, whether or not any exist.

Clustering#

K-means partitions data into k groups by repeatedly assigning points to the nearest centre and moving the centres. It is fast and assumes clusters are roughly spherical and similarly sized, which is often not true.

The algorithm is four steps and worth knowing exactly, because every one of its failure modes is visible in them. Place k centres at random points. Assign each data point to the nearest centre. Move each centre to the mean of the points assigned to it. Repeat the last two until nothing changes. Note what is missing: nothing anywhere in that loop asks whether the groups it is producing are real. It will terminate, and it will return k groups, on any data you give it.

from sklearn.cluster import KMeans

km = KMeans(n_clusters=5, n_init=10).fit(X)
km.labels_           # which cluster each point belongs to
km.inertia_          # total within-cluster distance, lower is tighter

Watch it invent clusters that are not there#

That note is the most important sentence in this section and it should not be taken on trust. Generate 500 points spread uniformly at random over a square — data with no groups in it by construction, the flattest possible null case — and run the whole standard procedure on it.

import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from sklearn.datasets import make_blobs

rng = np.random.default_rng(0)
noise = rng.uniform(0, 1, (500, 2))                     # no structure at all
blobs, _ = make_blobs(n_samples=500, centers=4, cluster_std=0.6, random_state=0)

for name, data in (("uniform", noise), ("blobs", blobs)):
    sil = []
    for k in range(2, 9):
        labels = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(data)
        sil.append(round(silhouette_score(data, labels), 3))
    print(name, sil)

# uniform [0.359, 0.384, 0.406, 0.385, 0.392, 0.389, 0.392]
# blobs   [0.556, 0.597, 0.684, 0.601, 0.498, 0.43,  0.313]

print(np.bincount(KMeans(n_clusters=4, n_init=10, random_state=0).fit_predict(noise)))
# [141 109 136 114]   four tidy, balanced groups, out of nothing

The elbow behaves no better. Inertia on the uniform data falls by factors of 1.62, 1.59 and 1.54 for the first three splits and then by only 1.22 — that bend is exactly what people point at and call an elbow at k equal to 4. On the real four-cluster data the same ratios are 2.41, 2.20 and 2.59, then 1.11. The difference is not whether there is a bend, it is how sharp the bend is, and reading sharpness off a plot by eye is how a whole team ends up with four customer segments that do not exist.

The advice usually offered next is to rerun with a different seed and check the clusters are stable. Check that too, because it also fails. Comparing runs from five different seeds with the adjusted Rand index gives 0.93 to 1.00 on the uniform data and exactly 1.00 on the real clusters. The noise partitions stably, because the partition is a property of the square's geometry rather than of any structure in the points.

What does work is comparison against a null. Generate random data with the same shape and range as yours, cluster it the same way, and compare the scores: 0.406 against 0.684 here. If your data's silhouette is not clearly above what pure noise of the same shape achieves, you have found nothing. That comparison, formalised, is the gap statistic, and it is the only one of these methods that asks the right question.

DBSCAN is worth knowing as the alternative: it finds clusters of arbitrary shape, decides the number itself, and labels sparse points as noise rather than forcing them into a group.

What the silhouette number actually is#

It is worth knowing what you are reading rather than treating it as a score out of one. For each point, take a, the average distance to the other points in its own cluster, and b, the average distance to the points in the nearest other cluster. The point's silhouette is b minus a, divided by whichever of the two is larger. It is 1 when the point's own cluster is far closer than any other, 0 when the two are equally far, and negative when the point is actually nearer to another cluster than its own. The reported score is the average over all points.

from sklearn.metrics import silhouette_score

points = np.array([[0.0, 0.0], [0.0, 1.0], [5.0, 0.0], [5.0, 1.0]])
pair_labels = np.array([0, 0, 1, 1])

# By hand, for the first point:
#   a = distance to its one neighbour [0, 1]                    = 1.0
#   b = mean distance to [5, 0] and [5, 1] = (5 + sqrt(26)) / 2 = 5.0495
#   silhouette = (b - a) / max(a, b) = 4.0495 / 5.0495          = 0.802
print(silhouette_score(points, pair_labels))       # 0.8019609728144304

That last point matters more than the number. Cluster the two interleaved crescents of the classic two-moons dataset correctly and the silhouette is low, because the crescents are long and the ends of one are closer to the other crescent than to their own far end. A method scoring well on silhouette is a method producing round blobs, which is a statement about k-means rather than about your data.

Dimensionality reduction#

• PCA: finds the directions of greatest variance and projects onto them. Linear, fast, and the projection is interpretable as a combination of features.
• t-SNE and UMAP: non-linear, for visualising high-dimensional data in two dimensions. Good at showing local structure.

Anomaly detection#

The third use of unlabelled data is finding what does not fit. It is the same machinery pointed the other way: instead of describing the groups, you describe what normal looks like and flag whatever sits far from it. The useful framing is that an anomaly is rarely an extreme value in one column. It is an unusual combination — an ordinary amount at an unusual hour with an unusual number of items — which is why a rule on a single column misses almost everything worth catching.

Two cautions apply from the start. Unusual is not the same as wrong: a flag is a request for someone to look, never a verdict, and presenting it as one is how these systems lose the trust of the people who have to act on them. And because nobody can review a thousand flags, the number to measure is how many of the top twenty were worth a human's time, which is precision at k rather than any overall rate.

Where this comes back#

These unsupervised techniques reappear constantly in LLM work under different names. Clustering embeddings to discover topics in a corpus. Reducing dimensions to visualise what your retrieval index looks like. Anomaly detection to find eval cases that are unlike everything else.

classical name            what it is called in LLM work
clustering                topic discovery over a month of user questions
clustering                grouping eval failures by kind so you fix a cause, not a case
dimensionality reduction  the two-dimensional plot of a retrieval index
near-duplicate detection  deduplicating pretraining data, and spotting eval leakage
anomaly detection         finding the odd inputs your eval set never anticipated
anomaly detection         drift monitoring: today's inputs against the validated ones