Create Spark DataFrame. Can not infer schema for type

by

SparkSession.createDataFrame, which is used under the hood, requires an RDD / list of Row/tuple/list/dict* or pandas.DataFrame, unless schema with DataType is provided. Try to convert float to tuple like this: 

myFloatRdd.map(lambda x: (x, )).toDF()

or even better:

from pyspark.sql import Row

row = Row("val") # Or some other column name
myFloatRdd.map(row).toDF()

To create a DataFrame from a list of scalars you’ll have to use SparkSession.createDataFrame directly and provide a schema***:

from pyspark.sql.types import FloatType

df = spark.createDataFrame([1.0, 2.0, 3.0], FloatType())

df.show()

## +-----+
## |value|
## +-----+
## |  1.0|
## |  2.0|
## |  3.0|
## +-----+

but for a simple range it would be better to use SparkSession.range:

from pyspark.sql.functions import col

spark.range(1, 4).select(col("id").cast("double"))

* No longer supported.

** Spark SQL also provides a limited support for schema inference on Python objects exposing __dict__.

*** Supported only in Spark 2.0 or later.

More Related Contents:

  1. How to add a constant column in a Spark DataFrame?
  2. Spark Dataframe distinguish columns with duplicated name
  3. Pyspark: Split multiple array columns into rows
  4. How do I add a new column to a Spark DataFrame (using PySpark)?
  5. How to change a dataframe column from String type to Double type in PySpark?
  6. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  7. Retrieve top n in each group of a DataFrame in pyspark
  8. Updating a dataframe column in spark
  9. Dividing complex rows of dataframe to simple rows in Pyspark
  10. Filter Pyspark dataframe column with None value
  11. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  12. How to explode multiple columns of a dataframe in pyspark
  13. Spark add new column to dataframe with value from previous row
  14. PySpark: multiple conditions in when clause
  15. Pyspark: Replacing value in a column by searching a dictionary
  16. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  17. Pivot String column on Pyspark Dataframe
  18. Filtering DataFrame using the length of a column
  19. Multiple condition filter on dataframe
  20. How to return a “Tuple type” in a UDF in PySpark?
  21. Spark Equivalent of IF Then ELSE
  22. How to perform union on two DataFrames with different amounts of columns in spark?
  23. Does spark predicate pushdown work with JDBC?
  24. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  25. Explode in PySpark
  26. Filtering a Pyspark DataFrame with SQL-like IN clause
  27. Passing a data frame column and external list to udf under withColumn
  28. Using UDF ignores condition in when
  29. Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
  30. When to cache a DataFrame?

Leave a Comment