The Titanic Survival dataset

This post is part I of a walkthough of how I built and improved my submission to the Titanic Machine Learning competition on Kaggle. The goal of the competition is to create a machine learning model that predicts which passengers survived the Titanic shipwreck.

In this post and the next, I will walk through the process of creating a machine learning classification model using the Titanic dataset, which provides various information on the passengers of the Titanic and their survival.

Part I covers data exploration, cleansing and transformation. At the end of this post, we’ll have a set of features ready to be fed into our machine learning models.

The dataset can be found on Kaggle. It is split into two group, the training set (train.csv) and the test set (test.csv).

The training set, which is meant to be used to build machine learning models, comes with the outcome (survived or not) for each passenger. The test set has the same features as the training set, apart from the outcome.

Loading and exploring the data

import pandas as pd
import numpy as np

import seaborn as sns
import matplotlib.pyplot as plt

The output of the cell below shows the first few lines of our training set. The dataset contains the following variables :

• ‘PassengerId’- a unique identifier for each passenger
• ‘Pclass’ - the passenger’s class on the ship (1st, 2nd or 3rd)
• ‘Name’
• ‘Sex’
• ‘Age’
• ‘SibSp’ - total number of siblings and spouse(s?) on the ship
• ‘Parch’ - total number of parents and children on the ship
• ‘Ticket’ - the passenger’s ticket number
• ‘Fare’
• ‘Cabin’
• ‘Embarked’ - the port from which the passenger embarked (Cherbourg, Queenstown, Southampton)

as well as the outcome ‘Survived’.

train_data = pd.read_csv('train.csv')
train_data.head()

test_data = pd.read_csv('test.csv')
test_data.head()

Using panda’s describe function shows us that 38% of passengers in the training data survived the Titanic. We can also see that the passengers’ ages range from 0.4 to 80. We can see some features with missing data, such as “Age”, “Cabin” and “Embarked”.

train_data.describe(include = "all")

Exploring correlation between different features of our data set and “Survival”

In this section, we explore every feature in our training set and check whether it is correlated to survival. This will help us decide whether to use it as a feature in our machine learning model, and whether we need to transform it beforehand.

We will start exploring the gender and age features - it is common knowledge that women and children were evacuated first from the sinking ship.

Gender

The below bar plot shows that around 75% of women on board survived, whereas only 19% of men did! It looks like women had a lot more chance of surviving the shipwreck, so gender would be a helpful features to predict surival.

women = train_data[train_data['Sex'] == "female"]
perc_women_survived = round(women['Survived'].sum()/len(women)*100)

men = train_data[train_data['Sex'] == "male"]
perc_men_survived = round(men['Survived'].sum()/len(men)*100)

title = f"{perc_women_survived}% of women passengers survived, \nwhereas only {perc_men_survived}% of men did"

sns.set_palette('Pastel1')
sns.set_style('whitegrid')
sns.barplot(data = train_data, x = "Sex", y = "Survived")
plt.title(title)
plt.show()

Age

The figure below shows the spead of age for passengers who survived and those who did not, for both men and women. We can see that the spread is quite different for men and women. Also, it looks like there are more children in the group who survived, and more older people in the group who did not survive.

ax = sns.violinplot(data = train_data, y = "Survived", x = "Age", hue = "Sex", orient= "h", split = True)

Passenger class

If you’ve seen the Titanic movie, you know that socioeconomic status played a role in deciding which passengers were given the priority to evacuate the ship. Survival was not only based on gender or age but also class as can be seen on the bar plot below: more than 60% of first class passngers survived the shipwreck, and no more than 25% of third class passengers did.

ax = sns.barplot(data = train_data, x = "Pclass", y = "Survived")

Fare

The price a passenger paid for their ticket is also an indicator of their socioeconomic status. The violin plot below shows that more people paid a higher rate for their ticket in the group who survived. Most fare amounts are on the lower side of the x axis, this is probably due to a small number of outliers.

plt.figure(figsize = (10, 6))
ax = sns.violinplot(data = train_data, y = "Survived", x = "Fare", orient= "h")

Relatives

The dataset provides information on the number of siblings, spouses, children and parents accompanying each passenger. Let us explore whether the number of relatives a passenger had on board affected their chance of survival. The bar plot shows that passengers accompanied by up to 3 relatives had a higher chance of survival than passengers travelling alone. The chance of survival decreases beyond this point.

relatives = train_data["SibSp"] + train_data["Parch"]

plt.figure(figsize = (12, 6))
ax = sns.barplot(x= relatives, y = train_data["Survived"])
plt.xlabel("Number of relatives")

Text(0.5, 0, 'Number of relatives')

Port of embarkation

One of the variables provided in the dataset in the port of embarkation “Embarked”, which takes one of three values: Cherbourg, Queenstown or Southampton. It looks like passengers who embarked from Cherbourg had a higher rate of survivals.

ax = sns.barplot(data = train_data, x = "Embarked", y = "Survived")

So far, we’ve explored all variables aside from PassengerId, Name, Ticket, and Cabin.

• PassengerId is unique to each passenger and won’t be of much use for our predictions.
• Name is also unique to each passenger, however, it also contains the person’s title, which could be correlated to the survival. We will later explore this information.
• Ticket
• Cabin field has lots of missing data, however, the cabin number looks like it has the deck information.

Filling out missing data

Let’s have a look at the missing data in each of the data sets. It looks like the “Cabin” variable has a huge number of missing entries in both datasets. “Age” comes next with quite a few missing values. We also have a couple of missing values for “Embarked” and “Fare”.

In this section we are going to deal with the missing data, discard the data we don’t need, and fill out the missing values with sensible data where relevant.

We are going to create an array containing both data sets, training and test, so we can perform the same operations on both.

pd.concat([train_data.isnull().sum(), test_data.isnull().sum()], axis=1).rename(columns = {0 : "Train_data", 1: "Test_data"})

combined_data = [train_data, test_data]

Age

Let’s start with age. We are going to compute the mean age for every gender and class combination, and use these values to fill out missing “Age” data - this will be stored in a new column “Age_full”.

for data_set in combined_data: 
    mean_age_age_class = data_set.groupby(["Sex", "Pclass"])["Age"].mean()
 
    data_set['Age_full'] = data_set.apply(lambda row : row['Age'] if not pd.isnull(row["Age"]) else   
                                        mean_age_age_class[row['Sex']][row['Pclass']] , axis = 1)
    data_set.drop(['Age'], axis = 1)
    
train_data.head(10)

Port of embarkation

We have a couple missing “Embarked” values in the training set. We are going to fill them out with the most common value, which happens to be “S” for Southampton.

for data_set in combined_data: 
   embarked_mode = data_set['Embarked'].mode()
   data_set['Embarked'] = data_set['Embarked'].fillna(embarked_mode[0])

Fare

We have one missing “Fare” value in the test set for a third class passenger who embarked at Southampton, which we are going to replace with the average fare for third class passengers who embarked there in the test set.

mean_fare_port_class = test_data.groupby(["Embarked", "Pclass"])["Fare"].mean()

test_data['Fare'] = test_data['Fare'].fillna(mean_fare_port_class["S"][3])

Cabin

As mentioned above, the “Cabin” variable has tons of missing values. We could completely drop this feature, however, it looks like cabin numbers have a letter which could be the deck number, or particular sections of the ship. Let’s extract this info and store it in a new “Deck” variable.

deck = {"A": 1, "B": 2, "C": 3, "D": 4, "E": 5, "F": 6, "G": 7, "U": 8, "T": 9}

for data_set in combined_data:
    data_set['Cabin'] = data_set['Cabin'].fillna("U0")
    data_set['Deck'] = data_set['Cabin'].apply(lambda x: x[0])
    
    data_set['Deck'] = data_set['Deck'].map(deck)
    data_set['Deck'] = data_set['Deck'].astype(int)

Transforming variables

Now that we are done dealing with missing data, we have 2 new columns: “Age_full” which replaces “Age”, and “Deck” that we’ve extracted from the “Cabin” data.

In the following section, we are going to transform the features we are using for our model into integer variables to feed into our models.

Numerical variables such as “Age” and “Fare” will be transformed into categorical variables by splitting the data in groups/intervals. We will then transform all our categorical variables into integer variables.

Age

Let’s start by splitting passengers into age groups and see the survival rate of different age groups. It looks like children (0-18) is the age group with the highest survival rate, whereas 70+ year olds have a survival rate around 15%.

for data_set in combined_data:
    data_set['Age_group'] = data_set['Age_full'].map(lambda x: 0 if (x >= 0) & (x < 18) else 
                                                 (1 if (x >= 18) & (x < 30) else (
                                                  2 if (x >= 30) & (x < 40) else (
                                                  3 if (x >= 40) & (x < 50) else (
                                                  4 if (x >= 50) & (x < 60) else (
                                                  5 if (x >= 60) & (x < 70) else (
                                                  6 if x >= 70 else "NaN")))))))

df = pd.DataFrame((train_data.groupby(["Age_group"])["Survived"].sum())*100/(train_data.groupby(["Age_group"])["Survived"].count()))

df['Not Survived'] = df["Survived"].apply(lambda x : 100 - x)

ax = plt.subplot()
plt.bar(range(len(df)), df["Survived"], label = "Survived")
plt.bar(range(len(df)), df["Not Survived"], bottom = df["Survived"], label = "Not Survived", alpha = 0.3)
plt.legend()
ax.set_xticks(range(0, 7))
ax.set_xticklabels(['0-18', '18-30', '30-40', '40-50', '50-60', '60-70', '70+'])
ax.set_yticks(range(0, 105, 10))
ax.set_yticklabels(str(x) + "%" for x in range(0, 105, 10))
plt.title('Percentage of survival in each age group')
plt.xlabel('Age groups')
plt.show()

Rate

We will now create a categorical variable out of the “Rate” variable. Using Pandas qcut, we’ll split rate values into 4 rate categories, each corresponding to a quartile of the variable. The bar plot shows that the higher the rate category (and hence the rate), the higher the survival rate.

fare_labels = [0, 1, 2, 3]

for data_set in combined_data:
    data_set['Fare_cat'] = pd.qcut(data_set['Fare'], 4, fare_labels)
    data_set['Fare_cat'] = data_set['Fare_cat'].astype(int)
    
train_data.head()

df = pd.DataFrame((train_data.groupby(["Fare_cat"])["Survived"].sum())*100/(train_data.groupby(["Fare_cat"])["Survived"].count()))

df['Not Survived'] = df["Survived"].apply(lambda x : 100 - x)

ax = plt.subplot()
plt.bar(range(len(df)), df["Survived"], label = "Survived")
plt.bar(range(len(df)), df["Not Survived"], bottom = df["Survived"], label = "Not Survived", alpha = 0.3)
plt.legend()
ax.set_xticks(range(0, 4))
ax.set_xticklabels(['Cat1', 'Cat2', 'Cat3', 'Cat4'])
ax.set_yticks(range(0, 105, 10))
ax.set_yticklabels(str(x) + "%" for x in range(0, 105, 10))
plt.title('Percentage of survival in each fare category')
plt.xlabel('Fare categories')
plt.show()

Names and titles

We have not explored yet the “Name” feature. As mentioned earlier, a passenger’s name is a unique value for each passenger and is hence not likely to give any indications about their survival. However, if we look closer, this variable also holds the passenger’s title.

The value counts of the extracted titles show that the most common titles are “Mr”, “Miss”, “Mrs” and Master”. Other titles are quite rare and will be all grouped under one fifth category.

for data_set in combined_data:
    data_set['Title'] = data_set['Name'].str.extract(' ([A-Za-z]+)\.')

df = pd.concat([train_data, test_data])

pd.DataFrame(df['Title'].value_counts()).reset_index()

titles = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Rare": 5}


for data_set in combined_data:
    data_set['Title'] = data_set['Title'].replace(['Lady', 'Countess','Capt', 'Col','Don', 'Dr','Major', 'Rev', 'Sir', 'Dona', 'Jonkheer'], 'Rare')
    data_set['Title'] = data_set['Title'].replace(['Mlle', 'Ms'], 'Miss')
    data_set['Title'] = data_set['Title'].replace('Mme', 'Mrs')
 
    data_set['Title'] = data_set['Title'].map(titles)
    
    data_set['Title'] = data_set['Title'].fillna(0)
    data_set['Title'] = data_set['Title'].astype(int)
    
    data_set = data_set.drop(['Name'], axis=1)

Relatives, gender and ports of embarkation

In the below we create a new column “Relatives” which represents the total number of siblings, spouses, parents and children accompagniying a passenger, as well as a variable “Alone” which has either a value of 1 if the passenger had no relatives accompagnying them, or 0 if they did. We also map string values in columns “Sex” and “Embarked” to numerical values.

for data_set in combined_data:
    data_set['Relatives'] = data_set['SibSp'] + data_set['Parch']
    data_set['Alone'] = data_set['Relatives'].apply(lambda x : 1 if x == 0 else 0) 

gender = {'female' : 1, 'male': 0}

for data_set in combined_data:
    data_set['Sex'] = data_set['Sex'].map(gender)

ports = {'S' : 0, 'C': 1, 'Q' : 2}

for data_set in combined_data:
    data_set['Embarked'] = data_set['Embarked'].map(ports)
    data_set['Embarked'] = data_set['Embarked'].astype(int)
    
train_data.head()

We are now ready to build our models using the following features which all have integer values ‘Sex’, ‘Title’, ‘Age_group’, ‘Fare_cat’, ‘Pclass’, ‘Embarked’, ‘Deck’, ‘SibSp’, ‘Parch’, ‘Relatives’ and ‘Alone’. Stay tuned for part II of the Legendary Titanic where we’ll built different models and create a submission.