scala parallel collections degree of parallelism

by

With the newest trunk, using the JVM 1.6 or newer, use the:

collection.parallel.ForkJoinTasks.defaultForkJoinPool.setParallelism(parlevel: Int)

This may be a subject to changes in the future, though. A more unified approach to configuring all Scala task parallel APIs is planned for the next releases.

Note, however, that while this will determine the number of processors the query utilizes, this may not be the actual number of threads involved in running a query. Since parallel collections support nested parallelism, the actual thread pool implementation may allocate more threads to run the query if it detects this is necessary.

EDIT:

Starting from Scala 2.10, the preferred way to set the parallelism level is through setting the tasksupport field to a new TaskSupport object, as in the following example:

scala> import scala.collection.parallel._
import scala.collection.parallel._

scala> val pc = mutable.ParArray(1, 2, 3)
pc: scala.collection.parallel.mutable.ParArray[Int] = ParArray(1, 2, 3)

scala> pc.tasksupport = new ForkJoinTaskSupport(new scala.concurrent.forkjoin.ForkJoinPool(2))
pc.tasksupport: scala.collection.parallel.TaskSupport = scala.collection.parallel.ForkJoinTaskSupport@4a5d484a

scala> pc map { _ + 1 }
res0: scala.collection.parallel.mutable.ParArray[Int] = ParArray(2, 3, 4)

While instantiating the ForkJoinTaskSupport object with a fork join pool, the parallelism level of the fork join pool must be set to the desired value (2 in the example).

More Related Contents:

  1. Scala 2.8 breakOut
  2. Is the Scala 2.8 collections library a case of “the longest suicide note in history”? [closed]
  3. Why is Scala’s immutable Set not covariant in its type?
  4. What is the difference between JavaConverters and JavaConversions in Scala?
  5. When should I choose Vector in Scala?
  6. What does param: _* mean in Scala?
  7. Scala 2.8 collections design tutorial
  8. withFilter instead of filter
  9. What type to use to store an in-memory mutable data table in Scala?
  10. Scala Map implementation keeping entries in insertion order?
  11. Stream vs Views vs Iterators
  12. How to read files from resources folder in Scala?
  13. Function returns an empty List in Spark
  14. Scala: Why does Seq.contains take an Any argument, instead of an argument of the sequence type?
  15. Reduce, fold or scan (Left/Right)?
  16. Scala Sets contain the same elements, but sameElements() returns false
  17. Difference between MutableList and ListBuffer
  18. Scala: Parallel collection in object initializer causes a program to hang
  19. Scala best way of turning a Collection into a Map-by-key?
  20. Spark extracting values from a Row
  21. How to pass -D parameter or environment variable to Spark job?
  22. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  23. Scala: Merge map
  24. MatchError while accessing vector column in Spark 2.0
  25. Pattern matching on generic type in Scala
  26. Escape a dollar sign in string interpolation
  27. Why is it possible to declare variable with same name in the REPL?
  28. Explicit self-references with no type / difference with ”this”
  29. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  30. “return” and “try-catch-finally” block evaluation in scala

Leave a Comment