Populate row number in pyspark – Row number by Group

In order to populate row number in pyspark we use row_number() Function.  row_number() function along with partitionBy() of other column populates the row number by group. Let’s see an example on how to populate row number in pyspark and also we will look at an example of populating row number for each group.

  • Populate row number in pyspark – using row_number() function.
  • Populate row number in pyspark by group – using row_number() along with partitionBy() function.

We will be using the dataframe df_basket1

row number in pyspark 1

 

 

Populating Row number in pyspark:

Row number is populated by row_number() function. We will be using partitionBy(), orderBy() on a column so that row number will be populated. partitionBy() function does not take any argument as we are not grouping by any variable. As the result row number is populated and stored in the new column named “row_num” as shown below.

### Row number in pyspark

from pyspark.sql.window import Window
import pyspark.sql.functions as F
from pyspark.sql.functions import row_number

df_basket1 = df_basket1.select("Item_group","Item_name","Price", F.row_number().over(Window.partitionBy().orderBy(df_basket1['price'])).alias("row_num"))
df_basket1.show()

So the resultant row number populated dataframe in pyspark will be
row number in pyspark 2

 

 

Populate row number in pyspark by group

populate row number in pyspark row_number by group c1

Row number by group is populated by row_number() function. We will be using partitionBy() on a group, orderBy() on a column so that row number will be populated by group in pyspark. partitionBy() function takes the column name as argument on which we have to make the grouping . In our case grouping done on “Item_group” As the result row number is populated by “Item_group” and the result is stored in the new column named “row_num” as shown below.

### Row number in pyspark by group

from pyspark.sql.window import Window
import pyspark.sql.functions as F
from pyspark.sql.functions import row_number


df_basket1 = df_basket1.select("Item_group","Item_name","Price", F.row_number().over(Window.partitionBy(df_basket1['Item_group']).orderBy(df_basket1['price'])).alias("row_num"))
df_basket1.show()

So the resultant dataframe with row number populated by group will be
row number in pyspark 3

 


Other Related Topics:

 

Populate row number in pyspark – Row number by Group                                                                                          Populate row number in pyspark – Row number by Group

Author

  • Sridhar Venkatachalam

    With close to 10 years on Experience in data science and machine learning Have extensively worked on programming languages like R, Python (Pandas), SAS, Pyspark.

    View all posts