In this post, we will provide an example of the implementation of the ** K-Means** algorithm in python. This K-Means algorithm python example consists of clustering a dataset that contains information of all the stocks that compose the Standard & Poor Index.

This example contains the following five steps:

- Obtain the 500 tickers for the SPY & 500 by scrapping the tickers symbols from Wikipedia. The function
conduct this task.*obtain_parse_wike_snp500()* - Obtain closes prices from last year for each of the symbols using the Quandl API
- Calculate mean and variance of the returns for each stock
- Choose the best k value for the cluster the dataset
- Fit the model with the k number of cluster

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
import pandas as pd import numpy as np from math import ceil import bs4 import requests import quandl # need to do pip install quandl from sklearn.cluster import KMeans import matplotlib.pyplot as plt def obtain_parse_wiki_snp500(): """ Download and parse the Wikipedia list of S&P500 constituents using requests and Beautiful Soup. """ response = requests.get( "http://en.wikipedia.org/wiki/List_of_S%26P_500_companies" ) soup = bs4.BeautifulSoup(response.text) # This selects the first table, using CSS Selector syntax and then ignores the header row ([1:]) symbolslist = soup.select('table')[0].select('tr')[1:] # Obtain the symbol information for each row in the S&P500 constituent table symbols = [] for i, symbol in enumerate(symbolslist): tds = symbol.select('td') symbols.append( (tds[0].select('a')[0].text, # Ticker tds[1].select('a')[0].text, # Name tds[3].text # Sector ) ) return symbols tickers = obtain_parse_wiki_snp500() |

The **tickers **object is a list of tuples with the **ticker, company name and sector** of each company. To observe the structure of the tickers object we show the first 10 elements.

1 2 3 4 5 6 7 8 9 10 11 12 13 |
tickers[:10] [('MMM', '3M Company', 'Industrials'), ('ABT', 'Abbott Laboratories', 'Health Care'), ('ABBV', 'AbbVie Inc.', 'Health Care'), ('ABMD', 'ABIOMED Inc', 'Health Care'), ('ACN', 'Accenture plc', 'Information Technology'), ('ATVI', 'Activision Blizzard', 'Communication Services'), ('ADBE', 'Adobe Systems Inc', 'Information Technology'), ('AMD', 'Advanced Micro Devices Inc', 'Information Technology'), ('AAP', 'Advance Auto Parts', 'Consumer Discretionary'), ('AES', 'AES Corp', 'Utilities')] |

We will use this list in the following function to loop over each symbol of ** tickers** and get the close price from the Quandl API. We perform the loop using a try-except block to handle errors regarding symbols that were changed or are different in the Quandl database.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
def get_quandl_data(symbols): """ This function would loop over all the symbols from the SPY & 500 and retrieve the Close price column from the WIKI database from Quandl between the dates given by the start_date and end_date parameters """ symbols = [symbol[0] for symbol in symbols] stocks_info = [] for symbol in symbols: try: stockdata = quandl.get("WIKI/{}".format(symbol), start_date='2018-09-20', end_date='2019-09-20',column_index=4) print('Downloading data from symbol {}'.format(symbol)) stocks_info.append((symbol,stockdata.iloc[:,0])) except: print('The stock symbol {} is not on quandl database'.format(symbol)) return stocks_info data = get_quandl_data(tickers) |

The ** data** object is a list of tuples where each tuple has 2 elements. The first element is the ticker and the second element is the Date and Close Price for each of the stocks. We need to parse this information in order to make a dataframe that has the date as index and the close prices for each stock as columns. The following lines will do this job:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# Get only the symbols from the data object symbols = [data[i] for i in range(0,len(data))] # Get the date and prices from the data object (second element of the tuple) closes = [data[i][1] for i in range(0,len(data))] # Store closes object in a dataframe and obtain the transpose of the dataframe. #With this we will have the Date as index and the Prices of each stock as columns. closes = pd.DataFrame(closes).T # Rename column names with the symbols list closes.columns = symbols #Calculating annual mean returns and standard deviation of the returns returns = closes.pct_change().mean() * 252 std = closes.pct_change().std() * np.sqrt(252) #Concatenating the returns and variances into a single data-frame ret_var = pd.concat([returns, std], axis = 1).dropna() ret_var.columns = ["Returns","Standard Deviation"] |

The `ret_var`

dataframe has the following structure:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
ret_var.head(10) Returns Standard Deviation MMM 0.301229 0.122294 ABT 0.392332 0.138839 ABBV 0.457778 0.179314 ABMD 0.545361 0.242094 ACN 0.284499 0.131696 ATVI 0.596721 0.305587 ADBE 0.554549 0.216873 AMD 0.069001 0.585976 AAP -0.460970 0.394637 AES -0.052332 0.210608 |

In order to determine the optimal number of clusters ** k** for the

**dataset, we will fit different models of the**

*ret_var***algorithm while varying the**

*K-means***parameter in the range 2 to 14. For each model we calculate the**

*k***(SSE) by using the**

*Sum Squared Error***method of the model fitted. In each iteration we append the inertia to the**

*inertia_***list. Then we take the model with the less value of**

*sse***. (Inertia tells how far away the points within a cluster are. The small the inertia value is better.)**

*SSE*
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
#Converting ret_var into numpy array X = ret_var.values sse = [] for k in range(2,15): kmeans = KMeans(n_clusters = k) kmeans.fit(X) #SSE for each n_clusters sse.append(kmeans.inertia_) plt.plot(range(2,15), sse) plt.title("Elbow Curve") plt.show() |

The graph is names as ** Elbow Curve**, and shows that the optimal value of k is 5.

**Elbow Curve: Determine the optimal value of k. K-means Cluster**

We chose k=5 and fit the model with this parameter-value.

1 2 3 4 5 |
kmeans = KMeans(n_clusters = 5).fit(X) centroids = kmeans.cluster_centers_ plt.scatter(X[:,0],X[:,1], c = kmeans.labels_, cmap ="rainbow") plt.show() |

The different groups or cluster of the dataset are reflected in the following graph:

Clusters of the ret_var dataset. k=5

We can view the presence of outliers as only one point is on the upper right side of the graph. This outlier form its own cluster. In order to have a better categorization of the stocks within the SPY index, we would remove those stocks and fit the model another time.

1 2 3 4 5 6 7 8 9 10 11 12 13 |
# Find the stock with the highest value in the Standard Deviation variable stdOrder = ret_var.sort_values('Standard Deviation',ascending=False) first_symbol = stdOrder.index[0] # Drop the columns with the outliers values ret_var.drop(first_symbol,inplace=True) # Fit the model without the outliers X = ret_var.values kmeans =KMeans(n_clusters = 5).fit(X) centroids = kmeans.cluster_centers_ plt.scatter(X[:,0],X[:,1], c = kmeans.labels_, cmap ="rainbow") plt.show() |

**Clusters of ret_var dataset without Outliers. k=5**

The ** x axis** of the Figure 17, refers to the returns of the stocks and the

**is the standard deviation of each stock. So the stocks that are in the upper-right cluster are the stocks with the higher value of returns and standard deviation.**

*y axis*Finally we will assign to each stock it correspondent number of cluster(** 1,2,3,4,and 5**) and make a dataframe with this information. Having the information of cluster number for each stock, we can create a diversified portfolio in the long term, between stocks from different clusters.

1 2 3 4 5 |
stocks = pd.DataFrame(ret_var.index) # the dataframe structure allow concatenation cluster_labels = pd.DataFrame(kmeans.labels_) stockClusters = pd.concat([stocks, cluster_labels],axis = 1) stockClusters.columns = ['Symbol','Cluster'] |

The structure of the ** stockClusters** dataframe is the following:

These are the first rows of the ** stockClusters** dataframe. We conclude this section with a categorization of each stock from the SPY & 500 in terms of returns and risk. This could be an important tool for portfolio diversification. This concludes our K-Means algorithm python example.

## Leave a Reply