AttributeError: ‘DataFrame’ object has no attribute ‘map’

by

You can’t map a dataframe, but you can convert the dataframe to an RDD and map that by doing spark_df.rdd.map(). Prior to Spark 2.0, spark_df.map would alias to spark_df.rdd.map(). With Spark 2.0, you must explicitly call .rdd first.

More Related Contents:

  1. Calling Java/Scala function from a task
  2. Spark RDD to DataFrame python
  3. Convert spark DataFrame column to python list
  4. How to create a custom Estimator in PySpark
  5. Spark mllib predicting weird number or NaN
  6. Finding duplicates from large data set using Apache Spark
  7. How to find median and quantiles using Spark
  8. Spark Dataframe distinguish columns with duplicated name
  9. How to change dataframe column names in pyspark?
  10. I can’t seem to get –py-files on Spark to work
  11. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  12. How to add third-party Java JAR files for use in PySpark
  13. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. How to extract an element from a array in pyspark
  17. PySpark: multiple conditions in when clause
  18. Create Spark DataFrame. Can not infer schema for type
  19. How to transform data with sliding window over time series data in Pyspark
  20. Create single row dataframe from list of list PySpark
  21. Build a hierarchy from a relational data-set using Pyspark
  22. How do I convert an array (i.e. list) column to Vector
  23. Add column sum as new column in PySpark dataframe
  24. Pyspark – converting json string to DataFrame
  25. How to add suffix and prefix to all columns in python/pyspark dataframe
  26. Spark DAG differs with ‘withColumn’ vs ‘select’
  27. How to drop all columns with null values in a PySpark DataFrame?
  28. Pyspark changing type of column from date to string
  29. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe
  30. Random numbers generation in PySpark

Leave a Comment