Add Hours, minutes and seconds to timestamp in Pyspark

In order to add hours, minutes and seconds to timestamp in pyspark we will be using expr() function with interval in hours , minutes and seconds respectively. expr() Function with interval N hours add hours to timestamp in pyspark. expr() Function with interval of N minutes add minutes to timestamp in pyspark. expr() Function with interval N seconds add seconds to timestamp in pyspark Let’s see an Example for each.

  • Add hours to timestamp in pyspark
  • Add minutes to timestamp in pyspark
  • Add seconds to timestamp in pyspark
  • Add hours, minutes and seconds together in pyspark

First lets create the dataframe as shown below

###### create dataframe in pyspark

import pyspark.sql.functions as F
from datetime import datetime
data = [
  ('George', datetime(2010, 3, 24, 3, 19, 58), 4),
  ('Andrew', datetime(2009, 12, 12, 17, 21, 30), 5),
  ('Micheal', datetime(2010, 11, 22, 13, 29, 40), 2),
  ('Maggie', datetime(2010, 2, 8, 3, 31, 23), 8),
  ('Ravi', datetime(2009, 1, 1, 4, 19, 47), 2),
  ('Xien', datetime(2010, 3, 2, 4, 33, 51), 3),
]

df = sqlContext.createDataFrame(data, ['name', 'birthdaytime', 'grad_Score'])
df.show(truncate=False)

We will be using the dataframe named df

Add Hours, minutes and seconds to timestamp in Pyspark 1

 

 

Add hour to timestamp in pyspark

To Add hour to timestamp in pyspark we will be using expr() function and mentioning the interval inside it. ‘INTERVAL N HOURS’. expr() function takes interval in hours / minutes / seconds as argument. in the below case expr() function takes interval in hours as argument.

### Add hour to timestamp in pyspark

import pyspark.sql.functions as F

df1 = df.withColumn('birthdaytime_new', df.birthdaytime + F.expr('INTERVAL 2 HOURS'))
df1.show(truncate=False)

In our example to birthdaytime column we will be adding interval of 2 hours. So the resultant dataframe will be

Add Hours, minutes and seconds to timestamp in Pyspark 12

 

 

Add minutes to timestamp in pyspark

To Add minutes to timestamp in pyspark we will be using expr() function and mentioning the interval inside it. ‘INTERVAL N MINUTES’. expr() function takes interval in hours / minutes / seconds as argument. in the below case expr() function takes interval in minutes as argument.

### Add minutes to timestamp in pyspark

import pyspark.sql.functions as F

df1 = df.withColumn('birthdaytime_new', df.birthdaytime + F.expr('INTERVAL 50 minutes'))
df1.show(truncate=False)

In our example to birthdaytime column we will be adding interval of 50 minutes. So the resultant dataframe will be

Add Hours, minutes and seconds to timestamp in Pyspark 13

 

 

Add seconds to timestamp in pyspark

To Add seconds to timestamp in pyspark we will be using expr() function and mentioning the interval inside it. ‘INTERVAL N SECONDS’. expr() function takes interval in hours / minutes / seconds as argument. in the below case expr() function takes interval in seconds as argument.

### Add seconds to timestamp in pyspark

import pyspark.sql.functions as F

df1 = df.withColumn('birthdaytime_new', df.birthdaytime + F.expr('INTERVAL 30 seconds'))
df1.show(truncate=False)

In our example to birthdaytime column we will be adding interval of 30 seconds. So the resultant dataframe will be

Add Hours, minutes and seconds to timestamp in Pyspark 14

 

 

 

Add hours, minutes and seconds together in pyspark :

To Add hours, minutes and seconds together to timestamp in pyspark we will be using expr() function and mentioning the interval inside it. ‘INTERVAL N HOURS N MINUTES N SECONDS’. expr() function takes interval in hours / minutes / seconds as argument. in the below case expr() function takes interval in hours minutes and seconds together as argument.

### Add minutes hours seconds together in pyspark

df1 = df.withColumn('birthdaytime_new', df.birthdaytime + F.expr('INTERVAL 1 HOURS 30 MINUTES 20 SECONDS'))
df1.show(truncate=False)

In our example to birthdaytime column we will be adding interval of 1 hours 30 minutes and 20 seconds. So the resultant dataframe will be

Add Hours, minutes and seconds to timestamp in Pyspark 15

for more details you can refer this article

Other Related Topics:

 

Add Hours, minutes and seconds to timestamp in Pyspark                                                                                                    Add Hours, minutes and seconds to timestamp in Pyspark

Author

  • Sridhar Venkatachalam

    With close to 10 years on Experience in data science and machine learning Have extensively worked on programming languages like R, Python (Pandas), SAS, Pyspark.

    View all posts