‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark

by

toDF method is a monkey patch executed inside SparkSession (SQLContext constructor in 1.x) constructor so to be able to use it you have to create a SQLContext (or SparkSession) first:

# SQLContext or HiveContext in Spark 1.x
from pyspark.sql import SparkSession
from pyspark import SparkContext

sc = SparkContext()

rdd = sc.parallelize([("a", 1)])
hasattr(rdd, "toDF")
## False

spark = SparkSession(sc)
hasattr(rdd, "toDF")
## True

rdd.toDF().show()
## +---+---+
## | _1| _2|
## +---+---+
## |  a|  1|
## +---+---+

Not to mention you need a SQLContext or SparkSession to work with DataFrames in the first place.

More Related Contents:

  1. Convert pyspark string to date format
  2. How to split Vector into columns – using PySpark
  3. How to find median and quantiles using Spark
  4. How to use JDBC source to write and read data in (Py)Spark?
  5. Spark Dataframe distinguish columns with duplicated name
  6. Pyspark: Split multiple array columns into rows
  7. How to change dataframe column names in pyspark?
  8. Retrieve top n in each group of a DataFrame in pyspark
  9. Pyspark: explode json in column to multiple columns
  10. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  11. How to use a Scala class inside Pyspark
  12. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  13. Filter Pyspark dataframe column with None value
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. How to extract an element from a array in pyspark
  17. PySpark: multiple conditions in when clause
  18. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  19. Create Spark DataFrame. Can not infer schema for type
  20. Load CSV file with PySpark
  21. Create single row dataframe from list of list PySpark
  22. Cannot find col function in pyspark
  23. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  24. Concatenate two PySpark dataframes
  25. How do I convert an array (i.e. list) column to Vector
  26. Filtering DataFrame using the length of a column
  27. Add column sum as new column in PySpark dataframe
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. how to add Row id in pySpark dataframes [duplicate]
  30. Random numbers generation in PySpark

Leave a Comment