Function returns an empty List in Spark

by

It happens because filesInZip is not shared between workers. foreach operates on a local copy of filesInZip and when it finishes this copy is simply discarded and garbage collected. If you want to keep the results you should use transformation (most likely a flatMap) and return collected aggregated values.

def listFiles(stream: PortableDataStream): TraversableOnce[String] = ???

zipInputStream.flatMap(listFiles)

You can learn more from Understanding closures

More Related Contents:

  1. How to store custom objects in Dataset?
  2. Is the Scala 2.8 collections library a case of “the longest suicide note in history”? [closed]
  3. How to avoid duplicate columns after join?
  4. Flattening Rows in Spark
  5. Split 1 column into 3 columns in spark scala
  6. Provide schema while reading csv file as a dataframe
  7. How to flatmap a nested Dataframe in Spark
  8. Joining Spark dataframes on the key
  9. Scala Map implementation keeping entries in insertion order?
  10. How to read files from resources folder in Scala?
  11. Modify collection inside a Spark RDD foreach
  12. how to filter out a null value from spark dataframe
  13. Is there a reason not to use SparkContext.getOrCreate when writing a spark job?
  14. What are the various join types in Spark?
  15. Provide schema while reading csv file as a dataframe in Scala Spark
  16. Replace missing values with mean – Spark Dataframe
  17. How to calculate the size of dataframe in bytes in Spark?
  18. Why does partition parameter of SparkContext.textFile not take effect?
  19. Can I zip more than two lists together in Scala?
  20. Cross product in Scala
  21. Spark Scala list folders in directory
  22. Exception in thread “main” java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)
  23. Spark-Monotonically increasing id not working as expected in dataframe?
  24. How to remove parentheses around records when saveAsTextFile on RDD[(String, Int)]?
  25. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  26. How to sort an RDD in Scala Spark?
  27. Difference between two rows in Spark dataframe
  28. Run a scala code jar appear NoSuchMethodError:scala.Predef$.refArrayOps
  29. Spark UDF with varargs
  30. How to transpose an RDD in Spark

Leave a Comment