How to change a dataframe column from String type to Double type in PySpark?

by

There is no need for an UDF here. Column already provides cast method with DataType instance :

from pyspark.sql.types import DoubleType

changedTypedf = joindf.withColumn("label", joindf["show"].cast(DoubleType()))

or short string:

changedTypedf = joindf.withColumn("label", joindf["show"].cast("double"))

where canonical string names (other variations can be supported as well) correspond to simpleString value. So for atomic types:

from pyspark.sql import types 

for t in ['BinaryType', 'BooleanType', 'ByteType', 'DateType', 
          'DecimalType', 'DoubleType', 'FloatType', 'IntegerType', 
           'LongType', 'ShortType', 'StringType', 'TimestampType']:
    print(f"{t}: {getattr(types, t)().simpleString()}")

BinaryType: binary
BooleanType: boolean
ByteType: tinyint
DateType: date
DecimalType: decimal(10,0)
DoubleType: double
FloatType: float
IntegerType: int
LongType: bigint
ShortType: smallint
StringType: string
TimestampType: timestamp

and for example complex types

types.ArrayType(types.IntegerType()).simpleString()   

'array<int>'

types.MapType(types.StringType(), types.IntegerType()).simpleString()

'map<string,int>'

More Related Contents:

  1. How to add a constant column in a Spark DataFrame?
  2. Spark Dataframe distinguish columns with duplicated name
  3. Pyspark: Split multiple array columns into rows
  4. How do I add a new column to a Spark DataFrame (using PySpark)?
  5. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  6. Retrieve top n in each group of a DataFrame in pyspark
  7. Updating a dataframe column in spark
  8. Dividing complex rows of dataframe to simple rows in Pyspark
  9. Filter Pyspark dataframe column with None value
  10. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  11. How to explode multiple columns of a dataframe in pyspark
  12. Spark add new column to dataframe with value from previous row
  13. PySpark: multiple conditions in when clause
  14. Pyspark: Replacing value in a column by searching a dictionary
  15. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  16. Create Spark DataFrame. Can not infer schema for type
  17. Pivot String column on Pyspark Dataframe
  18. Filtering DataFrame using the length of a column
  19. Multiple condition filter on dataframe
  20. How to return a “Tuple type” in a UDF in PySpark?
  21. Load CSV file with Spark
  22. Does spark predicate pushdown work with JDBC?
  23. Explode in PySpark
  24. Rename nested field in spark dataframe
  25. Passing a data frame column and external list to udf under withColumn
  26. Best way to get the max value in a Spark dataframe column
  27. Apache Spark Python Cosine Similarity over DataFrames
  28. GroupBy column and filter rows with maximum value in Pyspark
  29. Concatenate two PySpark dataframes
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment