value toDF is not a member of org.apache.spark.rdd.RDD

now i found the reason, you should define case class in the object and outof the main function. look at here

Ok, I finally fixed the issue. 2 things needed to be done:

  1. Import implicits: Note that this should be done only after an instance of org.apache.spark.sql.SQLContext is created. It should be written as:

    val sqlContext= new org.apache.spark.sql.SQLContext(sc)

    import sqlContext.implicits._

  2. Move case class outside of the method: case class, by use of which you define the schema of the DataFrame, should be defined outside of the method needing it. You can read more about it here:

Leave a Comment