Hierarchical Clustering in Python

Hierarchical Clustering is an unsupervised machine learning technique used to group similar objects into clusters. It creates a hierarchy of clusters, which is represented as a tree-like structure called a dendrogram. There are two main types of hierarchical clustering:

1. Agglomerative Clustering (bottom-up approach)

2. Divisive Clustering (top-down approach)

In this example, we’ll focus on Agglomerative Clustering, which is more commonly used.

Steps for Hierarchical Clustering

1. Prepare the data: You'll need a dataset to work with.

2. Calculate the distance matrix: Measure how far apart each data point is from others.

3. Apply Agglomerative Clustering: Combine clusters iteratively.

4. Visualize the dendrogram: A dendrogram is a tree-like diagram that helps visualize the clusters.

Libraries You Need:

• scipy: For hierarchical clustering and dendrogram visualization.

• matplotlib: For plotting the dendrogram.

• sklearn: For clustering and dataset creation.

Code Implementation:

Let’s implement Agglomerative Hierarchical Clustering using Python.

import numpy as npimport matplotlib.pyplot as pltfrom sklearn.datasets import make_blobsfrom sklearn.cluster import AgglomerativeClusteringfrom scipy.cluster.hierarchy import dendrogram, linkage# Step 1: Prepare the data (using make_blobs to generate synthetic data)X, _ = make_blobs(n_samples=50, centers=3, random_state=42)# Step 2: Compute the linkage matrix for hierarchical clusteringZ = linkage(X, method='ward')# Step 3: Plot the dendrogramplt.figure(figsize=(10, 7))dendrogram(Z)plt.title("Dendrogram for Hierarchical Clustering")plt.xlabel("Samples")plt.ylabel("Euclidean Distance")plt.show()# Step 4: Apply Agglomerative Clustering# Perform agglomerative clustering with 3 clustersagg_clust = AgglomerativeClustering(n_clusters=3, affinity='euclidean', linkage='ward')y_hc = agg_clust.fit_predict(X)# Step 5: Visualize the clustersplt.figure(figsize=(8, 6))plt.scatter(X[:, 0], X[:, 1], c=y_hc, cmap='viridis', marker='o')plt.title("Agglomerative Clustering")plt.xlabel("Feature 1")plt.ylabel("Feature 2")plt.show()

Explanation of the Code:

1. Data Generation:

   • We use make_blobs() to generate synthetic 2D data with 3 centers (clusters).

2. Linkage Matrix:

   • We compute the linkage matrix Z using the linkage() function from scipy.cluster.hierarchy. This matrix represents the hierarchical clustering of the data.

   • The method='ward' argument specifies the use of Ward's method, which minimizes the variance within clusters.

3. Dendrogram:

   • The dendrogram() function from scipy.cluster.hierarchy is used to visualize the hierarchical clustering as a tree. It helps determine the number of clusters by cutting the dendrogram at a specific level.

4. Agglomerative Clustering:

   • We perform Agglomerative Clustering using AgglomerativeClustering() from sklearn.cluster.

   • n_clusters=3 specifies that we want to form 3 clusters.

   • affinity='euclidean' means we are using Euclidean distance as the metric for clustering.

   • linkage='ward' ensures that Ward’s method is used for agglomeration.

5. Cluster Visualization:

   • We plot the clusters using a scatter plot, where each data point is colored according to its assigned cluster.

Output:

1. Dendrogram: A plot that shows the hierarchical structure of the clusters. The horizontal lines represent the merging of clusters, and the vertical lines represent the distance between merged clusters.

   • By cutting the dendrogram at a specific height, you can choose the number of clusters.

2. Agglomerative Clustering Visualization: A scatter plot showing the clusters after applying agglomerative clustering. Each point is assigned to one of the 3 clusters.

Key Concepts in Hierarchical Clustering

• Agglomerative Clustering (Bottom-up approach):

  1. Each point starts as its own cluster.

  2. The closest clusters are merged at each step until all points are grouped into a single cluster.

• Dendrogram:

  • The dendrogram helps you decide the optimal number of clusters by cutting the tree at a specific level.

• Linkage Methods:

  • ward: Minimizes the variance of the merged clusters.

  • complete: Uses the maximum distance between points in clusters.

  • average: Uses the average distance between points in clusters.

  • single: Uses the minimum distance between points in clusters.

Advantages of Hierarchical Clustering

• Does not require the number of clusters to be specified in advance (for agglomerative clustering, the number of clusters can be decided after analyzing the dendrogram).

• Easy to understand and interpret.

• Can capture clusters with complex shapes.

Disadvantages

• Computationally expensive for large datasets.

• Sensitive to noise and outliers.

• Difficult to scale to large datasets compared to other clustering algorithms like K-Means.

Alternative: Divisive Clustering

While Agglomerative Clustering is the most common form of hierarchical clustering, Divisive Clustering is a top-down approach. However, it is not implemented in scikit-learn directly, and it’s less commonly used in practice compared to agglomerative clustering.

Conclusion

Hierarchical clustering is a powerful and intuitive clustering technique, especially for small datasets or when the number of clusters is not known in advance. By using dendrograms, you can visualize the merging of clusters and decide the appropriate number of clusters for your data. The agglomerative approach is widely used, and Python’s scikit-learn and scipy libraries make implementing hierarchical clustering simple and efficient.

Let me know if you need further assistance or if you'd like to explore other clustering methods!