call of distinct and map together throws NPE in spark library

by

Spark does not support nested RDDs or user-defined functions that refer to other RDDs, hence the NullPointerException; see this thread on the spark-users mailing list.

It looks like your current code is trying to group the elements of d by value; you can do this efficiently with the groupBy() RDD method:

scala> val d = sc.parallelize(Seq("Hello", "World", "Hello"))
d: spark.RDD[java.lang.String] = spark.ParallelCollection@55c0c66a

scala> d.groupBy(x => x).collect()
res6: Array[(java.lang.String, Seq[java.lang.String])] = Array((World,ArrayBuffer(World)), (Hello,ArrayBuffer(Hello, Hello)))

More Related Contents:

  1. Encoder error while trying to map dataframe row to updated row
  2. How to define schema for custom type in Spark SQL?
  3. Specifying the filename when saving a DataFrame as a CSV [duplicate]
  4. NullPointerException in Scala Spark, appears to be caused be collection type?
  5. What does setMaster `local[*]` mean in spark?
  6. Spark unionAll multiple dataframes
  7. How can I pass extra parameters to UDFs in Spark SQL?
  8. How to split a dataframe into dataframes with same column values?
  9. Spark performance for Scala vs Python
  10. How to use regex to include/exclude some input files in sc.textFile?
  11. How to define a custom aggregation function to sum a column of Vectors?
  12. How to write spark streaming DF to Kafka topic
  13. Stackoverflow due to long RDD Lineage
  14. Better way to convert a string field into timestamp in Spark
  15. Apache Spark how to append new column from list/array to Spark dataframe
  16. How to find spark RDD/Dataframe size?
  17. Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]
  18. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  19. Encode an ADT / sealed trait hierarchy into Spark DataSet column
  20. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  21. Replace null values in Spark DataFrame
  22. Spark RDD default number of partitions
  23. Why does Spark RDD partition has 2GB limit for HDFS?
  24. Calculate Cosine Similarity Spark Dataframe
  25. Spark : how to run spark file from spark shell
  26. Spark Dataframe Random UUID changes after every transformation/action
  27. How to set hadoop configuration values from pyspark
  28. Spark DataFrames when udf functions do not accept large enough input variables
  29. Filter spark DataFrame on string contains
  30. Spark textFile vs wholeTextFiles

Leave a Comment