Populate current date and current timestamp in pyspark

In order to populate current date and current timestamp in  pyspark we will be using current_date() and current_timestamp() function respectively. current_date() function populates current date in a column in pyspark. current_timestamp() function populates current timestamp in a column in pyspark Let’s see an Example for each.

  • Get current date in pyspark – populate current date in pyspark column
  • Get current timestamp in pyspark – populate timestamp in pyspark column

We will be using the dataframe named df

Get current date and current timestamp in pyspark – Populate current datetime in column 1

 

 

Get current date in pyspark – populate current date in pyspark column

Syntax:

 current_date()

current_date() function is used to gets the current date in pyspark. The current date is stored in the new column named “current_date” and thereby we will have the dataframe with current date populated in a new column.

### Get current date in pyspark – populate current date in pyspark column

from pyspark.sql.functions import current_date

df1 = df.withColumn("current_date",current_date())
df1.show()

Current date is populated and appended to the dataframe, so the resultant dataframe will be
Get current date and current timestamp in pyspark – Populate current datetime in column 2

 

 

 

Get current timestamp in pyspark – populate current timestamp in pyspark column

Syntax:

 current_timestamp()

current_timestamp() gets the current time in pyspark. The current timestamp is stored in the new column named “current_time” and thereby we will have the dataframe with current datetime (timestamp) populated in a new column.

### Get current timestamp in pyspark- populate current timestamp in pyspark column

from pyspark.sql.functions import current_timestamp

df1 = df.withColumn("current_time",current_timestamp())
df1.show(truncate=False)

Current date time is populated and appended to the dataframe, so the resultant dataframe will beGet current date and current timestamp in pyspark – Populate current datetime in column 3

 


Other Related Topics:

 

Populate current date and current timestamp in pyspark                                                                                                    Populate current date and current timestamp in pyspark

Author

  • Sridhar Venkatachalam

    With close to 10 years on Experience in data science and machine learning Have extensively worked on programming languages like R, Python (Pandas), SAS, Pyspark.

    View all posts