Scala spark, listbuffer is empty

by

Apache Spark doesn’t provide shared memory therefore here:

dataSet.foreach { e =>
  items += e
  println("len = " + items.length) //1. here length is ok
}

you modify a local copy of items on a respective exectuor. The original items list defined on the driver is not modified. As a result this:

items.foreach { x => print(x) }

executes, but there is nothing to print.

Please check Understanding closures

While it would be recommended here, you could replace items with an accumulator

val acc = sc.collectionAccumulator[String]("Items")
dataSet.foreach(e => acc.add(e))

More Related Contents:

  1. Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only
  2. How MapReduce work in Apache Spark and Scala?
  3. Write to multiple outputs by key Spark – one Spark job
  4. How does HashPartitioner work?
  5. Spark – load CSV file as DataFrame?
  6. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  7. How to define partitioning of DataFrame?
  8. Case class equality in Apache Spark
  9. Why accesing DataFrame from UDF results in NullPointerException?
  10. SparkSQL: How to deal with null values in user defined function?
  11. How to aggregate values into collection after groupBy?
  12. Dropping a nested column from Spark DataFrame
  13. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  14. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  15. Spark / Scala: forward fill with last observation
  16. Change nullable property of column in spark dataframe
  17. Reading DataFrame from partitioned parquet file
  18. Perform a typed join in Scala with Spark Datasets
  19. Apache Spark: Get number of records per partition
  20. Left Anti join in Spark?
  21. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  22. DataFrame-ified zipWithIndex
  23. Derive multiple columns from a single column in a Spark DataFrame
  24. How to process multi line input records in Spark
  25. How to explode an array into multiple columns in Spark
  26. Why is join not possible after show operator?
  27. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  28. How to create DataFrame from Scala’s List of Iterables?
  29. How to create a Dataset of Maps?
  30. Spark ML VectorAssembler returns strange output

Leave a Comment