Re arrange or re order column in pyspark

In order to Re arrange or re order the column in pyspark we will be using select function. To reorder the column in ascending order we will be using Sorted function. To reorder the column in descending order we will be using Sorted function with an argument reverse =True. lets get clarity with an example.

  • Re arrange the column in pyspark
  • Re order the column in pyspark in ascending order
  • Re order the column in pyspark in descending order

We will use the dataframe named df_basket1.

Re arrange or re order column in pyspark 1

 

 

Re arrange the column in pyspark :

Using select function in pyspark we can select the column in the order which we want which in turn re arranges the column according to the order that we want which is shown below

df_basket_reordered = df_basket1.select("price","Item_group","Item_name")
df_basket_reordered.show()

so the resultant dataframe with rearranged columns will be

Re arrange or re order column in pyspark 2

 

 

Re order the column in pyspark in ascending order

With the help of select function along with the sorted function in pyspark we first sort the column names in ascending order. Column name is passed to the sorted () function and then it is selected using select function as shown below.

## Re order column by ascending order
df_basket_reordered = df_basket1.select(sorted(df_basket1.columns))
df_basket_reordered.show()

So the resultant dataframe with columns sorted in ascending order will be

Re arrange or re order column in pyspark 3

 

 

Re order the column in pyspark in descending order

Column name is passed to the sorted () function along with the argument reverse=True which sorts the column in descending order and then it is selected using select function as shown below.

## Re order column by descending order
df_basket_reordered = df_basket1.select(sorted(df_basket1.columns,reverse=True))
df_basket_reordered.show()

Re arrange or re order column in pyspark 4

 

Re arrange or re order column in pyspark                                                                                             Re arrange or re order column in pyspark