value toDF is not a member of org.apache.spark.rdd.RDD

by

now i found the reason, you should define case class in the object and outof the main function. look at here

Ok, I finally fixed the issue. 2 things needed to be done:

  1. Import implicits: Note that this should be done only after an instance of org.apache.spark.sql.SQLContext is created. It should be written as:

    val sqlContext= new org.apache.spark.sql.SQLContext(sc)

    import sqlContext.implicits._

  2. Move case class outside of the method: case class, by use of which you define the schema of the DataFrame, should be defined outside of the method needing it. You can read more about it here: https://issues.scala-lang.org/browse/SI-6649

More Related Contents:

  1. How to use sbt from behind proxy?
  2. How to access a secured Nexus with sbt?
  3. How to set main class in build?
  4. How to Add Environment Profile Config to SBT
  5. How to share version values between project/plugins.sbt and project/Build.scala?
  6. How to do aggregate functions, may columns and extract back
  7. How to pivot Spark DataFrame?
  8. How to melt Spark DataFrame?
  9. Load CSV file with Spark
  10. Encoder error while trying to map dataframe row to updated row
  11. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  12. Difference between df.repartition and DataFrameWriter partitionBy?
  13. How can I change column types in Spark SQL’s DataFrame?
  14. How to check if spark dataframe is empty?
  15. Does SparkSQL support subquery?
  16. How to connect to remote hive server from spark [duplicate]
  17. Spark DataFrame: does groupBy after orderBy maintain that order?
  18. How to construct Dataframe from a Excel (xls,xlsx) file in Scala Spark?
  19. Spark UDAF with ArrayType as bufferSchema performance issues
  20. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  21. How to use SQL query to define table in dbtable?
  22. Apache Spark Python Cosine Similarity over DataFrames
  23. Run JUnit tests with SBT
  24. Spark Error:expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)
  25. How do I add an persistent column of row ids to Spark DataFrame?
  26. Debugging Scala code with simple-build-tool (sbt) and IntelliJ
  27. Convert null values to empty array in Spark DataFrame
  28. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  29. How to suppress info and success messages in sbt?
  30. Calculate Cosine Similarity Spark Dataframe

Leave a Comment