Exploring customers’ personalities using Python

In this tutorial, we will be exploring a dataset containing customers’ information such as Marital Status, Education, Income, etc. Getting an insight into the kinds of customers a firm deals with helps to provide better and customized services. Therefore, customer analysis and segmentation play a crucial role in a business.

The outline for this tutorial is as follows:

1. About the Dataset
2. Importing the necessary libraries
3. Exploratory Data Analysis
4. Plotting the Correlation
5. Encoding the data
6. Scaling the data
7. Dimensionality reduction
8. CLustering
9. Visualizing the clusters
10. Observation
11. Conclusion

1. About the Dataset

The dataset used for this tutorial is publicly available on Kaggle. The total number of instances (or rows) are 2240 whereas the total number of attributes (or columns) is 29. As mentioned above, each attribute corresponds to a person trait important for customer classification such as Marital Status, Income, Customer ID, etc.

Go ahead and download the dataset! We will next be importing libraries to begin our analysis.

2. Importing the necessary libraries

import numpy as np
import pandas as pd
import datetime
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import colors
import seaborn as sns
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import AgglomerativeClustering
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.colors import ListedColormap
from sklearn import metrics
from sklearn.preprocessing import LabelEncoder
from sklearn.decomposition import PCA

In case you run into a package installation error, you can install it by typing pip install package-name in the command line. Alternatively, if you’re using a conda virtual environment, you can type conda install package-name.

Loading our dataset and Exploratory Data Analysis (EDA)

Let’s go ahead and load our CSV file.

Using GridDB

To store large amounts of data, a CSV file can be cumbersome and chaotic. GridDB serves as a great alternative as it is open-source and a highly scalable database. It is optimized for IoT and Big Data so that you can store your time-series data effectively. It utilizes an in-memory data architecture along with parallel processing for higher performance. You can Download GridDB from their official website.

By using the GridDB-Python client, we can load our data directly into the Python environment as a dataframe.

import griddb_python as griddb
import pandas as pd

sql_statement = ('SELECT * FROM marketing_campaign')
movie_review_test = pd.read_sql_query(sql_statement, cont)

Note that the cont variable has the container information where our data is stored. Replace the marketing_campaign with the name of your container. You can also refer to the tutorial on reading and writing to GridDB for more information.

Using Pandas

If you do not have GridDB, you can load the file using Pandas.

data = pd.read_csv('marketing_campaign.csv', sep="\t")

We’ll display the first five rows using the head command to get a gist of our data.

data.head()

data.head()

len(data)

2240

As mentioned earlier, we have a total of 2240 instances. Let’s eliminate any missing values since they can behave abnormally during mathematical operations.

data = data.dropna()
len(data)

2216

We had 24 missing instances in our dataset. Let’s also convert the Date attribute of our dataset to the DateTime format as it will later be used for calculating the time a customer has been active.

data["Dt_Customer"] = pd.to_datetime(data["Dt_Customer"])
dates = []
for i in data["Dt_Customer"]:
    i = i.date()
    dates.append(i) 

We will be creating a new attribute – Customer_dur to calculate the duration for which a customer has been associated with the firm. For the sake of simplicity, we will be extracting the date of the most recent customer and using it to calculate the duration for others.

days = []
d1 = max(dates) 
for i in dates:
    delta = d1 - i
    days.append(delta)
data["Customer_dur"] = days
data["Customer_dur"] = pd.to_numeric(data["Customer_dur"], errors="coerce")

Great! Now let’s have a look at the next attribute – Marital Status. From the head command, it seems like there are more than two categories.

data["Marital_Status"].value_counts()

Married     857
Together    573
Single      471
Divorced    232
Widow        76
Alone         3
Absurd        2
YOLO          2
Name: Marital_Status, dtype: int64

There are 8 further categories of this attribute. However, we will be grouping them together to ease out our segmentation process. We will be creating a binary attribute – Living_With having either of the two values – Partner or Alone.

data["Living_With"]=data["Marital_Status"].replace({"Married":"Partner", "Together":"Partner", "Absurd":"Alone", "Widow":"Alone", "YOLO":"Alone", "Divorced":"Alone", "Single":"Alone",})

Now that we have a derivative feature, Marital_Status serves as a redundant attribute. Let’s go ahead and drop that. We’ll also be dropping some columns that contain promotions’ and deals’ information as it is not relevant for customer segmentation.

cols_del = ['Marital_Status', 'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'AcceptedCmp1','AcceptedCmp2', 'Complain', 'Response']
data = data.drop(cols_del, axis=1)

Let’s have a look at how many categories does the Education attribute hold.

data["Education"].value_counts()

Graduation    1116
PhD            481
Master         365
2n Cycle       200
Basic           54
Name: Education, dtype: int64

We will repeat the same step with the Education attribute as well. In this case, the categories will be Undergraduate, Graduate, Postgraduate.

data["Education"]=data["Education"].replace({"Basic":"Undergraduate","2n Cycle":"Undergraduate", "Graduation":"Graduate", "Master":"Postgraduate", "PhD":"Postgraduate"})

We will be creating some new features derived from the original ones for easy calculations in the later sections. These features are as follows:

1. Age: can be derived from the Year_of_Birth.
2. Spent: sum of all the edible products (wine, fruits, fish, etc.).
3. Children: sum of Kidhome and Teenhome.
4. Family_Size: We’ll use the Living_With attribute along with the Children attribute.
5. Is_Parent: Binary attribute with values 0 or 1 derived from the Children attribute.

data["Age"] = 2021-data["Year_Birth"]
data["Spent"] = data["MntWines"]+ data["MntFruits"]+ data["MntMeatProducts"]+ data["MntFishProducts"]+ data["MntSweetProducts"]+ data["MntGoldProds"]
data["Children"]=data["Kidhome"]+data["Teenhome"]
data["Family_Size"] = data["Living_With"].replace({"Alone": 1, "Partner":2})+ data["Children"]
data["Is_Parent"] = np.where(data.Children> 0, 1, 0)

Like previously, we’ll drop the redundant attributes.

to_drop = ["Dt_Customer", "Z_CostContact", "Z_Revenue", "Year_Birth", "ID"]
data = data.drop(to_drop, axis=1)

After all the changes, our data looks like –

data.describe()

4. Plotting the correlation

We will now plot these newly created features pair-wise while using the Is_Parent as the main classification attribute.

To_Plot = [ "Income", "Age", "Spent", "Is_Parent"]
plt.figure()
sns.pairplot(data[To_Plot], hue= "Is_Parent") 
plt.show()

<Figure size 432x288 with 0 Axes>

We can see that we have a few outliers such as:

1. Age>100 is highly unlikely. This implies that some of the data instances might be old. Also, the majority of the data is concentrated below the threshold of 80. So, we’ll eliminate the few instances that cross that threshold

2. Income>600000 has only one instance.

We’ll go ahead and delete that.

data = data[(data["Age"]<90)]
data = data[(data["Income"]<600000)]

len(data)

2212

5. Encoding the data

Categorical variables need to be encoded before a machine learning task as the string value can not be directly used in mathematical operations. Let’s print out the categorical attributes in our dataset.

s = (data.dtypes == 'object')
object_cols = list(s[s].index)

object_cols

['Education', 'Living_With']

We will use the scikit-learn’s Label Encoder which encodes the target values into classes: 0 to n-1. In our case, Education has three categories, so the values will be denoted by 0,1,2 respectively. Similarly, Living_With is a binary attribute, therefore, the values will take the form of 0 or 1.

LE=LabelEncoder()
for i in object_cols:
    data[i]=data[[i]].apply(LE.fit_transform)

data.head()

6. Scaling the data

It is also important to note that these numerical values differ in scale. This could lead to a biased model which gives much significance to one attribute over the other. Therefore, it is important to map them on a similar scale.

Standard Scaler removes the mean and scales the features to unit variance. More information on how unit variance is calculated can be found here.

scaler = StandardScaler()
scaler.fit(data)
scaled_data = pd.DataFrame(scaler.transform(data),columns= data.columns)

scaled_data.head()

7. Dimensionality reduction

We now have the scaled data but the total number of columns is still large to deal with. A higher number of columns lead to higher dimensions which makes it harder to work with. For the sake of simplicity, we will be reducing the total number of columns to 3 as some of them are already redundant.

Principal Component Analysis (PCA) is one of the popular techniques used for dimensionality reduction as it minimizes information loss.

pca = PCA(n_components=3)
pca.fit(scaled_data)
pca_data = pd.DataFrame(pca.transform(scaled_data), columns=(["c1","c2", "c3"]))

pca_data.describe().T

8. Clustering

Now that our data has been cleaned and reduced to 3 dimensions, we can go ahead and divide it into clusters. For this tutorial, we will be using Agglomerative Clustering from the scikit-learn library. Agglomerative Clustering is a recursive method of clustering that uses linkage distance. More information can be found here.

The number of clusters is chosen as 4. However, for real-world models, there are approaches such as the Elbow method to anticipate the ideal number of clusters for a sample of data.

ac = AgglomerativeClustering(n_clusters=4)
customer_ac = ac.fit_predict(pca_data)

Let’s add this data to our original dataframes.

pca_data["Clusters"] = customer_ac
data["Clusters"]= customer_ac

9. Visualizing the clusters

Great! It is now time to plot our clusters and see how they look in 3D.

x =pca_data["c1"]
y =pca_data["c2"]
z =pca_data["c3"]

fig = plt.figure(figsize=(8,8))
ax = plt.subplot(111, projection='3d')
ax.scatter(x, y, z, s=20, c=pca_data["Clusters"])
plt.show()

We will now plot the pairwise plots of the Spent attribute with the other attributes we created previously in this tutorial. This will give us an insight as to how the final results are affected by each attribute and what kind of customers belong to each cluster.

columns = [ "Customer_dur", "Age", "Family_Size", "Is_Parent", "Education","Living_With"]
h = data["Clusters"]
for col in columns:
    plt.figure()
    sns.jointplot(x=data[col], y=data["Spent"], hue=h, kind="kde")
    plt.show()

C:\Users\Shripriya\anaconda3\lib\site-packages\seaborn\distributions.py:434: UserWarning: The following kwargs were not used by contour: 'hue'
  cset = contour_func(xx, yy, z, n_levels, **kwargs)



<Figure size 432x288 with 0 Axes>

C:\Users\Shripriya\anaconda3\lib\site-packages\seaborn\distributions.py:434: UserWarning: The following kwargs were not used by contour: 'hue'
  cset = contour_func(xx, yy, z, n_levels, **kwargs)



<Figure size 432x288 with 0 Axes>

C:\Users\Shripriya\anaconda3\lib\site-packages\seaborn\distributions.py:434: UserWarning: The following kwargs were not used by contour: 'hue'
  cset = contour_func(xx, yy, z, n_levels, **kwargs)



<Figure size 432x288 with 0 Axes>

C:\Users\Shripriya\anaconda3\lib\site-packages\seaborn\distributions.py:434: UserWarning: The following kwargs were not used by contour: 'hue'
  cset = contour_func(xx, yy, z, n_levels, **kwargs)



<Figure size 432x288 with 0 Axes>

C:\Users\Shripriya\anaconda3\lib\site-packages\seaborn\distributions.py:434: UserWarning: The following kwargs were not used by contour: 'hue'
  cset = contour_func(xx, yy, z, n_levels, **kwargs)



<Figure size 432x288 with 0 Axes>

C:\Users\Shripriya\anaconda3\lib\site-packages\seaborn\distributions.py:434: UserWarning: The following kwargs were not used by contour: 'hue'
  cset = contour_func(xx, yy, z, n_levels, **kwargs)



<Figure size 432x288 with 0 Axes>

10. Observation

Some key insights from the above joint plots:

1. Cluster 0: Certainly a parent, 40-70 age group, family size of 2-4.
2. Cluster 1: Not a parent, 30-80 age group, family size of 1-2.
3. Cluster 2: The majority of customers are a parent, all age groups from 20-80, family size of 1-3.
4. Cluster 3: Certainly a parent, 35-75 age group, family size of 2-5.

The Customer_dur attribute spans all across the clusters and is not specific to one, therefore we get a widely spread shaped.

Note: The seaborn library is not updated to its latest version on the system where the tutorial was executed and therefore, some features like contour did not behave as expected. However, if you have an updated environment, the above graphs will be much clearer with distinct lines and cluster labels.

11. Conclusion

In this tutorial, we explored customer profiles and see how it can affect a firm’s business. We also segmented the customers using Agglomerative Clustering. In the end, we identified some key features corresponding to each cluster. Finally, we introduced an alternative for storing large amounts of data in a highly efficient and scalable way – GridDB!

This code has been inspired from Kaggle

If you have any questions about the blog, please create a Stack Overflow post here https://stackoverflow.com/questions/ask?tags=griddb .
Make sure that you use the “griddb” tag so our engineers can quickly reply to your questions.