Check and Count Missing values in pandas python

isnull() is the function that is used to check missing values or null values in pandas python. isna() function is also used to get the count of missing values of column and row wise count of missing values.In this Section we will look at how to check and count Missing values in pandas python. And also group by count of missing values of a column.Let’s get started with below list of examples

• is there any missing values in dataframe as a whole
• is there any missing values across each columnf
• count of missing values across each column using isna() and isnull()
• count row wise missing value using isnull().
• count of  missing values of a specific column.
• groupby count of missing values of a column.

Let’s first create the dataframe.

#Create a DataFrame

import pandas as pd
import numpy as np

df1 = {
    'Name':['George','Andrea','micheal','maggie','Ravi','Xien','Jalpa',np.nan],
    'State':['Arizona','Georgia','Newyork','Indiana','Florida','California',np.nan,np.nan],
    'Gender':["M","F","M","F","M","M",np.nan,np.nan],      
    'Score':[63,48,56,75,np.nan,77,np.nan,np.nan]
    
   }


df1 = pd.DataFrame(df1,columns=['Name','State','Gender','Score'])
print(df1)

So the resultant dataframe will be

Let’s check is there any missing values in dataframe as a whole

'''' is any missing values in dataframe '''
df1.isnull()

output:

Let’s check is there any missing values across each column

''' is any missing values across columns'''
df1.isnull().any()

There are  missing values in all the columns

output:

Get count of missing values of the entire dataframe in pandas:

In order to get the count of missing values of the entire dataframe we will be using isnull().sum() which does the column wise sum first and doing another sum() will get the count of missing values of the entire dataframe

''' count of missing values of the entire dataframe'''

df1.isnull().sum().sum()

so the count of missing values of the entire dataframe will be

Get count of Missing values of each column in pandas python: Method 1

In order to get the count of missing values of each column in pandas we will be using isnull() and sum() function as shown below

''' count of missing values column wise'''
df1.isnull().sum()

So the column wise missing values of all the column will be

output:

Get count of Missing values of each column in pandas python: Method 2

In order to get the count of missing values of each column in pandas we will be using isna() and sum() function as shown below

''' count of missing values across columns'''
df1.isna().sum()

So the column wise missing values of all the column will be

output:

Get count of Missing values of each column in pandas python: Method 3

In order to get the count of missing values of each column in pandas we will be using len() and count() function as shown below

''' count of missing values across columns'''

count_nan = len(df1) - df1.count()
count_nan

So the column wise missing values of all the column will be

output:

Get count of Missing values of rows  in pandas python: Method 1

In order to get the count of row wise missing values in pandas we will be using isnull() and sum() function with axis =1 represents the row wise operations as shown below

''' count of missing values across rows'''

df1.isnull().sum(axis = 1)

So the row wise count of  missing values will be

output:

Get count of Missing values of rows  in pandas python: Method 2

In order to get the count of row wise missing values in pandas we will be using isnull() and sum() function with for loop which performs the row wise operations as shown below

''' count of missing values across rows'''

for i in range(len(df1.index)) :
    print("Nan in row ", i , " : " ,  df1.iloc[i].isnull().sum())

So the row wise count of missing values will be

output:

count of missing values of a particular column in pandas:

In order to get the count of missing values  of the particular column in pandas we will be using isnull() and sum() function with for loop which gets the count of missing values of a particular column as shown below

### get count of missing values of a particular column

df1.Gender.isnull().sum()

So the  count of missing values of particular column will be

output:

count of missing values of a column by group:

In order to get the count of missing values  of the particular column by group in pandas we will be using isnull() and sum() function with apply() and groupby() which performs the group wise count of missing values as shown below

### get count of missing values of a particular column by group

df1.groupby(['Gender'])['Score'].apply(lambda x: x.isnull().sum())

So the  count of missing values of “Score” column by group (“Gender”) will be

for further details on missing data kindly refer here