Visualizing Missing Data
Revisiting a simple skill with some new tricks
Missing Data
Dealing with missing values in data is an inevitability in data science. Finding those missing values and determining what to do with them will make or break any project. At flatiron learning how to find missing values is one of the first methods we learn.
df.isnull.()sum()This simple line of code is a standard when taking your initial look through your data. It finds all the missing values and returns a sum of all the missing values. This is great, but what if you want to know the more specifics about the missing values? Are there patterns in the missing data? Is there a specific column or row with a disproportionate amount of missing data? In this blog, I will show some code I have been using to help visualize missing values and help analyze the data.
Plot it
The first visualization is super simple. Just plot out the isnull function. Plotting out the missing values this way will help give a quick look at the missing values and how it is distributed throughout the data.
sns.heatmap(series.isnull(),yticklabels = False, cbar = False,cmap = 'tab20c_r')
plt.title('Missing Data: Series')
plt.show()
Here is a heat-map depicting the distribution of missing values in a series of vaccination data. As you can see with just a few simple lines of code there are a lot of missing values. This is a neat trick but what if we want to get a better feel for just how many missing values there are in this visualization.
Quantifying the missing values
The next step is my new favorite method for really seeing just how many missing values there are in your data.
rows = series.shape[0]
null_total = series.isnull().sum()missing_percent = (null_total/rows)*100
missing_percent
With this code you can find the percentage of missing values
Now we can see the percentages of missing data for each column in the dataset. The only downside is that this list isn’t the most interpretable. To create a better visualization, these percentages can be turned into a dataframe
pd.DataFrame(missing_percent ,columns = ["missing_percent"])
Now we have a nice neat table that shows us the percentages of missing values for each column.
These are some simple tricks for some initial data analysis. I hope that you find this code useful. It is most defiantly one of my favorite new coding tricks that I will be using in all future projects.
