Drop column in pyspark – drop single & multiple columns

Deleting or Dropping column in pyspark can be accomplished using drop() function. drop() Function with argument column name is used to drop the column in pyspark.  We will see how to

  • Drop single column in pyspark with example
  • Drop multiple column in pyspark with example
  • Drop column like function in pyspark – drop similar column

We will be using df.

Drop column in pyspark – drop single & multiple columns 1

 

 

Drop single column in pyspark – Method 1 :

Drop single column in pyspark using drop() function. Drop function with the column name as argument drops that particular column.

## drop single column

df.drop('mathematics_score').show()

So the resultant dataframe has mathematics_score column dropped

Drop column in pyspark – drop single & multiple columns 2

 

 

Drop single column in pyspark –  Method 2:

Drop single column in pyspark using drop() function. Drop function with the df.column_name as argument drops that particular column.

## drop single column

df.drop(df.mathematics_score).show()

So the resultant dataframe has mathematics_score column dropped

Drop column in pyspark – drop single & multiple columns 3

 

 

Drop multiple column in pyspark :

Drop multiple column in pyspark using drop() function. Drop function with list of column names as argument drops those columns.

## drop multiple columns

df.drop('mathematics_score','science_score').show()

So the resultant dataframe has mathematics_score and science_score columns dropped

Drop column in pyspark – drop single & multiple columns 4

 

Drop column in pyspark – drop single & multiple columns                                                                                                Drop column in pyspark – drop single & multiple columns