Exploratory data analysis¶
The MovieLens 1M dataset¶
This is a dataset of movie ratings data collected from users of MovieLens in the late 1990s and early 2000s. The data provide movie ratings, movie metadata, and demographic data about the users. Such data is often of interest in the development of recommendation systems based on machine learning algorithms.
The MovieLens 1M dataset contains ~1 million ratings collected from ~6,000 users on ~4,000 movies. It's spread across three tables: ratings, user information, and movie information. We can load each table into a pandas DataFrame object using the Python code below.
See: https://grouplens.org/datasets/movielens/ for additional information.
# Imports
import numpy as np
import pandas as pd
from pandas import DataFrame, Series
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import pearsonr
import pandas as pd
# Make display smaller
pd.options.display.max_rows = 10
unames = ['user_id', 'gender', 'age', 'occupation', 'zip']
users = pd.read_csv('movielens/movielens/users.dat', sep='::',
header=None, names=unames, engine='python')
rnames = ['user_id', 'movie_id', 'rating', 'timestamp']
ratings = pd.read_csv('movielens/movielens/ratings.dat', sep='::',
header=None, names=rnames, engine='python')
mnames = ['movie_id', 'title', 'genres']
movies = pd.read_csv('movielens/movielens/movies.dat', sep='::',
header=None, names=mnames, engine='python')
Next we write Python code to answer the following questions:
- How many users are stored in the users table and what information is stored for each user?
- How many movies are stored in the movies table and what information is stored for each movie?
- How many ratings are stored in the ratings table and what information is stored for each rating?
- How are users, the movies each user has rated, and the rating related?
Note: ages and occupations are coded as integers indicating groups described in the dataset’s README file.
print("Number of users: ", len(users))
users
print("Number of movies: ", len(movies))
movies
print("Number of ratings:", len(ratings))
ratings
There are 6040 users in the users table. Each user has a unique user ID, gender, age, occupation, and zip code.
There are 3883 movies in the movies table. Each movie has a unique movie ID, genre, and title.
There are 1000209 ratings in the ratings table. Each rating is mapped to a user through the user ID, and a movie through the movie ID. The table also contains the rating itself as well as a timestamp of wheen the rating was made.
The ratings dataset allows us to uniquely identify which users made ratings through the user ID and exactly which movies those unique users reviewed through the movie ID.
Next we write Python code to answer the following questions:
- What is the occupation that maps to most of the users?
- What percentage of users are 50 years old or older?
- Which movie received the highest number of ratings (and how were such ratings distributed)?
- What is the average rating for all movies/users?
users['occupation'].value_counts().plot.bar()
plt.xlabel("Occupation")
plt.ylabel("Frequency")
plt.title("Occupation Frequencies")
plt.show()
occupations=np.array(users['occupation'].value_counts())
print("Size of most frequent occupation: ", np.max(occupations))
users['age'].value_counts().plot.pie(autopct='%1.0f%%')
plt.show()
print(ratings['movie_id'].value_counts())
print(np.where(movies['movie_id']==2858))
print(movies['title'][2789])
most_rated_movie_indices=np.array(np.where(ratings['movie_id']==2858))
most_rated_movie_ratings=[]
for index in most_rated_movie_indices:
most_rated_movie_ratings.append(ratings['rating'][index])
most_rated_movie_ratings=np.array(most_rated_movie_ratings)
print(most_rated_movie_ratings[0])
plt.boxplot(most_rated_movie_ratings[0])
plt.title("American Beauty Ratings")
plt.xlabel("Category")
plt.ylabel("Rating")
plt.show()
ratings_arr=np.array(ratings['rating'])
print("Average of all ratings: ", np.mean(ratings_arr))
Occupation 4 is the occupation that maps to most of the users. This corresponds to college/grad student, according to the README. It occurs 759 times.
14% of the users from the dataset are 50 years old or older. This is obtained by adding the percentages from age groups 50 and 56 in the pie charts below.
Movie 2858, or "American Beauty (1999)" received the highest number of ratings. Most ratings were distributed around the top, that is, 4s and 5s.
The average rating for all movies and users is approximately 3.58.
We will use the Python code below to merge all three tables into a unified data frame.
data = pd.merge(pd.merge(ratings, users), movies)
data
The Python code below will show the top 10 films among female viewers (and, for comparison's sake, the ratings for those movies by male viewers) in decreasing order (highest rated movie on top).
# Build pivot table
mean_ratings = data.pivot_table('rating', index='title',
columns='gender', aggfunc='mean')
# Group ratings by title
ratings_by_title = data.groupby('title').size()
# Select only movies with 250 ratings or more
active_titles = ratings_by_title.index[ratings_by_title >= 250]
# Select rows on the index
mean_ratings = mean_ratings.loc[active_titles]
# Fix naming inconsistency
mean_ratings = mean_ratings.rename(index={'Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954)':
'Seven Samurai (Shichinin no samurai) (1954)'})
top_female_ratings = mean_ratings.sort_values(by='F', ascending=False)
top_female_ratings[:10]
The Python code below will:
- Display the top 10 favorite movies among male viewers, selecting only movies with 250 ratings or more.
- Display the top 10 favorite movies among young viewers (17 years old or younger), selecting only movies with 300 ratings or more.
# Build pivot table
mean_ratings = data.pivot_table('rating', index='title',
columns='gender', aggfunc='mean')
# Group ratings by title
ratings_by_title = data.groupby('title').size()
# Select only movies with 250 ratings or more
active_titles = ratings_by_title.index[ratings_by_title >= 250]
# Select rows on the index
mean_ratings = mean_ratings.loc[active_titles]
# Fix naming inconsistency
mean_ratings = mean_ratings.rename(index={'Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954)':
'Seven Samurai (Shichinin no samurai) (1954)'})
top_male_ratings = mean_ratings.sort_values(by='M', ascending=False)
top_male_ratings[:10]
# Build pivot table
mean_ratings = data.pivot_table('rating', index='title',
columns='age', aggfunc='mean')
# Group ratings by title
ratings_by_title = data.groupby('title').size()
# Select only movies with 300 ratings or more
active_titles = ratings_by_title.index[ratings_by_title >= 300]
# Select rows on the index
mean_ratings = mean_ratings.loc[active_titles]
# Fix naming inconsistency
mean_ratings = mean_ratings.rename(index={'Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954)':
'Seven Samurai (Shichinin no samurai) (1954)'})
top_young_ratings = mean_ratings.sort_values(by=1, ascending=False)
top_young_ratings[:10]
Next we write Python code to display the most divisive movies (selecting only movies with 250 ratings or more), i.e.:
- The top 10 movies with the greatest rating difference so that we can see which ones were preferred by women.
- The top 10 movies with the greatest rating difference in the opposite direction (sign) so that we can see which ones were preferred by men.
# Build pivot table
mean_ratings = data.pivot_table('rating', index='title',
columns='gender', aggfunc='mean')
# Add difference column
mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']
# Group ratings by title
ratings_by_title = data.groupby('title').size()
# Select only movies with 250 ratings or more
active_titles = ratings_by_title.index[ratings_by_title >= 250]
# Select rows on the index
mean_ratings = mean_ratings.loc[active_titles]
# Fix naming inconsistency
mean_ratings = mean_ratings.rename(index={'Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954)':
'Seven Samurai (Shichinin no samurai) (1954)'})
top_female_male_ratings = mean_ratings.sort_values(by='diff', ascending=True)
top_female_male_ratings[:10]
# Build pivot table
mean_ratings = data.pivot_table('rating', index='title',
columns='gender', aggfunc='mean')
# Add difference column
mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']
# Group ratings by title
ratings_by_title = data.groupby('title').size()
# Select only movies with 250 ratings or more
active_titles = ratings_by_title.index[ratings_by_title >= 250]
# Select rows on the index
mean_ratings = mean_ratings.loc[active_titles]
# Fix naming inconsistency
mean_ratings = mean_ratings.rename(index={'Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954)':
'Seven Samurai (Shichinin no samurai) (1954)'})
top_male_female_ratings = mean_ratings.sort_values(by='diff', ascending=False)
top_male_female_ratings[:10]
Next we write Python code to display the top 10 movies (with 250 ratings or more) that elicited the most disagreement among viewers, independent of gender identification.
# Build pivot table
std_ratings = data.pivot_table('rating', index='title', aggfunc='std')
# Group ratings by title
ratings_by_title = data.groupby('title').size()
# Select only movies with 250 ratings or more
active_titles = ratings_by_title.index[ratings_by_title >= 250]
# Select rows on the index
std_ratings = std_ratings.loc[active_titles]
# Fix naming inconsistency
std_ratings = std_ratings.rename(index={'Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954)':
'Seven Samurai (Shichinin no samurai) (1954)'})
top_divisive_ratings = std_ratings.sort_values(by='rating', ascending=False)
top_divisive_ratings[:10]
Next we write Python code to answer the question:
What is the most popular movie genre?
from collections import defaultdict
g = movies['genres'].str.split("|")
tally = defaultdict(lambda: 0)
for list_ in g:
for item in list_:
tally[item] +=1
plt.figure(figsize=(20,5))
plt.bar(tally.keys(), tally.values())
plt.show()
So, according to the chart above, Drama is the most popular movie genre.
The Titanic Dataset¶
In this part we'll use the dataset of passengers on the Titanic, available through the Seaborn library.
See https://www.kaggle.com/c/titanic/data for codebook and additional information.
titanic = sns.load_dataset('titanic')
titanic.head()
Next we will use the Python code below to answer the following questions (expressing the amounts in % terms):
- How many female passengers did not survive (regardless of their class)?
- How many first class female passengers did not survive?
- How many male passengers did not survive (regardless of their class)?
- How many third class male passengers did not survive?
titanic.pivot_table('survived', index='sex', columns='class', margins=True)
sns.catplot(x="sex", y="survived", hue="class", kind="bar", data=titanic);
Approximately 25.8% of female passengers did not survive, regardless of their class.
Approximately 3.2% of female passengers in first class did not survive.
Approximately 81.2% of male passengers did not survive, regardless of their class.
Approximately 86.5% of male passengers in third class did not survive.
Next we write Python code to answer the following questions:
- How many passengers (absolute number) were there per deck/class?
- How many passengers (absolute number) in Deck A survived?
- How many passengers (absolute number) in Deck E survived and what was the breakdown (in Deck E) per class?
sns.countplot(y="deck", hue="class", data=titanic)
titanic.pivot_table('survived', index='deck', columns='class', aggfunc='count')
print("Surviving Deck A Passengers: \n")
titanic[titanic['deck'] == 'A'].pivot_table('survived', index='deck', columns='class', aggfunc='sum')
print("Surviving Deck E Passengers by class: \n")
titanic[titanic['deck'] == 'E'].pivot_table('survived', index='deck', columns='class', aggfunc='sum')
print("Class Breakdown in Deck E: \n")
titanic[titanic['deck'] == 'E'].pivot_table('survived', index='deck', columns='class', aggfunc='count')
Next we write Python code to answer the following questions (using percentage values):
- How many women traveling alone did not survive?
- How many men 35 years old or younger did not survive?
- What was the average fare per class?
print("Percentage of survivors: ")
titanic.pivot_table('survived', index='sex', columns='alone')
print("So, 21.5% of women traveling alone did not survive.")
young_men=titanic[[titanic['sex']=='male'] and titanic['age']<=35]
young_men.pivot_table('survived', index='deck', columns='class', margins=True)
print("So, 21.2% of men 35 years old or younger did not survive.")
print("Average fare per class: ")
titanic.pivot_table('fare', index='sex', columns='class', margins=True)
US Baby Names 1880–2018¶
The United States Social Security Administration (SSA) has made available data on the frequency of baby names from 1880 through the present. These plain text data files, one per year, contain the total number of births for each sex/name combination. The raw archive of these files can be obtained from http://www.ssa.gov/oact/babynames/limits.html.
In the 'names' folder, you will have a directory containing a series of files like yob1880.txt through yob2018.txt. We need to do some data wrangling to load this dataset (see code below).
years = range(1880, 2019)
pieces = []
columns = ['name', 'sex', 'births']
for year in years:
path = 'names/yob%d.txt' % year
frame = pd.read_csv(path, names=columns)
frame['year'] = year
pieces.append(frame)
# Concatenate everything into a single DataFrame
names = pd.concat(pieces, ignore_index=True)
names
Next we write Python code to compute the number of baby boys and baby girls born each year and display the two line plots over time:
births_by_gender=names.pivot_table('births', index='year', columns='sex', aggfunc='sum')
births_by_gender.plot.line()
plt.show()
Analyzing Naming Trends¶
Suppose we're interested in analyzing the Top 1000 most popular baby names per year.
We will do so by following these steps:
- Insert a column prop with the fraction of babies given each name relative to the total number of births. A prop value of 0.02 would indicate that 2 out of every 100 babies were given a particular name in a given year.
- Group the data by year and sex, then add the new column to each group.
- Extract a subset of the data (the top 1,000 names for each sex/year combination). This is yet another group operation.
- Split the Top 1,000 names into the boy and girl portions.
- Build a pivot table of the total number of births by year and name.
Finally, we will plot the percentage of babies named 'John', 'Noah', 'Madison', or 'Lorraine' over time.
def add_prop(group):
group['prop'] = group.births / group.births.sum()
return group
names = names.groupby(['year', 'sex']).apply(add_prop)
names
# Sanity check (all percentages should add up to 1, i.e., 100%)
names.groupby(['year', 'sex']).prop.sum()
def get_top1000(group):
return group.sort_values(by='births', ascending=False)[:1000]
grouped = names.groupby(['year', 'sex'])
top1000 = grouped.apply(get_top1000)
# Drop the group index, not needed
top1000.reset_index(inplace=True, drop=True)
top1000
boys = top1000[top1000.sex == 'M']
girls = top1000[top1000.sex == 'F']
total_births = top1000.pivot_table('births', index='year',
columns='name',
aggfunc=sum)
total_births.info()
subset = total_births[['John', 'Noah', 'Madison', 'Lorraine']]
subset.plot(subplots=True, figsize=(12, 10), grid=False,
title="Number of births per year")
plt.show()
Boy names that became girl names (and vice versa)¶
Next, let's look at baby names that were more popular with one sex earlier in the sample but have switched to the opposite sex over the years. One example is the name Lesley or Leslie (or other possible, less common, spelling variations).
We will do so by following these steps:
- Go back to the top1000 DataFrame and compute a list of names occurring in the dataset starting with “lesl”.
- Filter down to just those names and sum births grouped by name to see the relative frequencies.
- Aggregate by sex and year and normalize within year.
- Plot the breakdown by sex over time.
all_names = pd.Series(top1000.name.unique())
lesley_like = all_names[all_names.str.lower().str.contains('lesl')]
lesley_like
filtered = top1000[top1000.name.isin(lesley_like)]
filtered.groupby('name').births.sum()
table = filtered.pivot_table('births', index='year',
columns='sex', aggfunc='sum')
table = table.div(table.sum(1), axis=0)
fig = plt.figure()
table.plot(style={'M': 'b-', 'F': 'r--'})
plt.show()
Next we make hypothesis H2:
H2: The name 'Taylor' has become more prevalent among baby girls since 2000.
Next we write Python code to test hypothesis H2 (and some text to explain whether it was confirmed or not):
all_names = pd.Series(top1000.name.unique())
lesley_like = all_names[all_names.str.lower().str.contains('taylor')]
lesley_like
filtered = top1000[top1000.name.isin(lesley_like)]
filtered.groupby('name').births.sum()
table = filtered.pivot_table('births', index='year',
columns='sex', aggfunc='sum')
table = table.div(table.sum(1), axis=0)
table
fig = plt.figure()
table.plot(style={'M': 'b-', 'F': 'r--'})
plt.show()
So, H2 has been proven true by the plot above.