Python Playground

How can I select the characters or file path after the Dev\” and dev\ from the column in a spark D… Read more Pyspark: Split And Select Part Of The String Column Values

I am trying to measure the performance impact on having to copy a dataframe from scala to python an… Read more Pyspark. Transformer That Generates A Random Number Generates Always The Same Number

I am writing unit tests for a Spark job, and some of the outputs are named tuples: pyspark.sql.Row … Read more Check If Two Pyspark Rows Are Equal

I tried taking a schema as a common schema by df.schema() and load all the CSV files to it .But fai… Read more Spark - Merge / Union Dataframe With Different Schema (column Names And Sequence) To A Dataframe With Master Common Schema

I'm fairly new to stream programming. We have Kafka stream which use Avro. I want to connect a … Read more Spark Stream - 'utf8' Codec Can't Decode Bytes

I have created a Glue job that copies data from S3 (csv file) to Redshift. It works and populates … Read more Aws Glue Truncate Redshift Table

How to create a new column in PySpark and fill this column with the date of today? This is what I t… Read more Assertionerror: Col Should Be Column

I have a script with the below setup. I am using: 1) Spark dataframes to pull data in 2) Converting… Read more Convert A Pandas Dataframe To A Pyspark Dataframe

I am using spark 2.1.0. I am not able to create timestamp column in pyspark I am using below code s… Read more Pyspark Creating Timestamp Column

I was solving this example : https://www.windowfunctions.com/questions/grouping/5 Here, they use Or… Read more How To Get Postgres Command 'nth_value' Equivalent In Pyspark Hive Sql?

Let's say I have these two Numpy arrays: A = np.arange(1024 ** 2, dtype=np.float64).reshape(102… Read more Multiply Two Numpy Matrices In Pyspark

I'm new to Python and PySpark. I have a dataframe in PySpark like the following: ## +---+---+--… Read more Create A Column In A Pyspark Dataframe Using A List Whose Indices Are Present In One Column Of The Dataframe

+------------------------------------------------------------------+ | message … Read more Extract Json Data From A Spark Dataframe

I want to change List to Vector in pySpark, and then use this column to Machine Learning model for … Read more What Type Should The Dense Vector Be, When Using Udf Function In Pyspark?

I am trying to use data from a spark dataframe as the input for my k-means model. However I keep ge… Read more Pyspark - Valueerror: Could Not Convert String To Float / Invalid Literal For Float()

I'm trying to implement KMeans algorithm using Pyspark it gives me the above error in the last … Read more Typeerror: Object Of Type 'map' Has No Len() Python3

I'm working with Spark 1.3.0 using PySpark and MLlib and I need to save and load my models. I u… Read more What Is The Right Way To Save\load Models In Spark\pyspark

I am trying to overwrite a Spark dataframe using the following option in PySpark but I am not succe… Read more Overwriting A Spark Output Using Pyspark

I have the following dataframe: df = pd.DataFrame([['A', 1],['B', 2],['C', … Read more Assign A New Column In Pandas In A Similar Way As In Pyspark

In pyspark I have a column called test_time. This is a timestamp column. The column has a records l… Read more Adding 1 Hours To Time Stamp Columns In Pyspark Data Frame