Convert A Pandas Dataframe To A Pyspark Dataframe

I have a script with the below setup. I am using: 1) Spark dataframes to pull data in 2) Converting to pandas dataframes after initial aggregatioin 3) Want to convert back to Spark

Solution 1:

Here we go:

# Spark to Pandas
df_pd = df.toPandas()

# Pandas to Spark
df_sp = spark_session.createDataFrame(df_pd)