NullPointerException in Scala Spark, appears to be caused be collection type?

by

Spark doesn’t support nesting of RDDs (see https://stackoverflow.com/a/14130534/590203 for another occurrence of the same problem), so you can’t perform transformations or actions on RDDs inside of other RDD operations.

In the first case, you’re seeing a NullPointerException thrown by the worker when it tries to access a SparkContext object that’s only present on the driver and not the workers.

In the second case, my hunch is the job was run locally on the driver and worked purely by accident.

More Related Contents:

  1. How MapReduce work in Apache Spark and Scala?
  2. Write to multiple outputs by key Spark – one Spark job
  3. How does HashPartitioner work?
  4. Spark 2.0 Dataset vs DataFrame
  5. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  6. Automatically and Elegantly flatten DataFrame in Spark SQL
  7. Why accesing DataFrame from UDF results in NullPointerException?
  8. How to pass -D parameter or environment variable to Spark job?
  9. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  10. Append a column to Data Frame in Apache Spark 1.3
  11. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  12. Processing multiple files as independent RDD’s in parallel
  13. MatchError while accessing vector column in Spark 2.0
  14. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  15. Spark / Scala: forward fill with last observation
  16. How to convert unix timestamp to date in Spark
  17. Apache Spark: Get number of records per partition
  18. Derive multiple columns from a single column in a Spark DataFrame
  19. How to save a spark DataFrame as csv on disk?
  20. How to process multi line input records in Spark
  21. How to sort by column in descending order in Spark SQL?
  22. How to get keys and values from MapType column in SparkSQL DataFrame
  23. Spark : how to run spark file from spark shell
  24. Spark Dataframe Random UUID changes after every transformation/action
  25. How to create a Dataset of Maps?
  26. How to set hadoop configuration values from pyspark
  27. Spark DataFrames when udf functions do not accept large enough input variables
  28. Filter spark DataFrame on string contains
  29. Spark textFile vs wholeTextFiles
  30. Spark ML VectorAssembler returns strange output

Leave a Comment