Modify collection inside a Spark RDD foreach

by

It becomes clearer when running on a Spark cluster (not a single machine). The RDD is now spread over several machines. When you call foreach, you tell each machine what to do with the piece of the RDD that it has. If you refer to any local variables (like myMap), they get serialized and sent to the machines, so they can use it. But nothing comes back. So your original copy of myMap is unaffected.

I think this answers your question, but obviously you are trying to accomplish something and you will not be able to get there this way. Feel free to explain here or in a separate question what you are trying to do, and I will try to help.

More Related Contents:

  1. How does HashPartitioner work?
  2. How to convert rdd object to dataframe in spark
  3. (Why) do we need to call cache or persist on a RDD
  4. Case class equality in Apache Spark
  5. Spark performance for Scala vs Python
  6. Stackoverflow due to long RDD Lineage
  7. How to find spark RDD/Dataframe size?
  8. Parsing multiline records in Scala
  9. Explanation of fold method of spark RDD
  10. Why does partition parameter of SparkContext.textFile not take effect?
  11. Why does Spark RDD partition has 2GB limit for HDFS?
  12. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  13. How to sort an RDD in Scala Spark?
  14. How to transpose an RDD in Spark
  15. How to bucket the range of values from a column and count how many values fall into each interval in scala?
  16. How to query JSON data column using Spark DataFrames?
  17. How do I detect if a Spark DataFrame has a column
  18. How to define schema for custom type in Spark SQL?
  19. How to avoid duplicate columns after join?
  20. Flattening Rows in Spark
  21. Specifying the filename when saving a DataFrame as a CSV [duplicate]
  22. How to split a dataframe into dataframes with same column values?
  23. How to use regex to include/exclude some input files in sc.textFile?
  24. How to zip two (or more) DataFrame in Spark
  25. How to force DataFrame evaluation in Spark
  26. How to write spark streaming DF to Kafka topic
  27. call of distinct and map together throws NPE in spark library
  28. Encode an ADT / sealed trait hierarchy into Spark DataSet column
  29. Provide schema while reading csv file as a dataframe in Scala Spark
  30. Spark – Error “A master URL must be set in your configuration” when submitting an app

Leave a Comment