Converting Pandas dataframe into Spark dataframe error

by

I made this script, It worked for my 10 pandas Data frames

from pyspark.sql.types import *

# Auxiliar functions
def equivalent_type(f):
    if f == 'datetime64[ns]': return TimestampType()
    elif f == 'int64': return LongType()
    elif f == 'int32': return IntegerType()
    elif f == 'float64': return DoubleType()
    elif f == 'float32': return FloatType()
    else: return StringType()

def define_structure(string, format_type):
    try: typo = equivalent_type(format_type)
    except: typo = StringType()
    return StructField(string, typo)

# Given pandas dataframe, it will return a spark's dataframe.
def pandas_to_spark(pandas_df):
    columns = list(pandas_df.columns)
    types = list(pandas_df.dtypes)
    struct_list = []
    for column, typo in zip(columns, types): 
      struct_list.append(define_structure(column, typo))
    p_schema = StructType(struct_list)
    return sqlContext.createDataFrame(pandas_df, p_schema)

You can see it also in this gist

With this you just have to call spark_df = pandas_to_spark(pandas_df)

More Related Contents:

  1. Why is Apache-Spark – Python so slow locally as compared to pandas?
  2. How to zip two array columns in Spark SQL
  3. How to split Vector into columns – using PySpark
  4. Spark Dataframe distinguish columns with duplicated name
  5. Encode and assemble multiple features in PySpark
  6. How to change dataframe column names in pyspark?
  7. How to export a table dataframe in PySpark to csv?
  8. Retrieve top n in each group of a DataFrame in pyspark
  9. Pyspark: explode json in column to multiple columns
  10. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  11. How to use a Scala class inside Pyspark
  12. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  13. Filter Pyspark dataframe column with None value
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. PySpark: multiple conditions in when clause
  17. PySpark create new column with mapping from a dict
  18. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  19. Create Spark DataFrame. Can not infer schema for type
  20. Load CSV file with PySpark
  21. Using UDF ignores condition in when
  22. Create single row dataframe from list of list PySpark
  23. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  24. How to pass a constant value to Python UDF?
  25. When to cache a DataFrame?
  26. Multiple condition filter on dataframe
  27. How to return a “Tuple type” in a UDF in PySpark?
  28. How to connect HBase and Spark using Python?
  29. How to pivot on multiple columns in Spark SQL?
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment