Parallel stream from a HashSet doesn’t run in parallel

by

I can reproduce the behavior you see, where the parallelism doesn’t match the parallelism of the fork-join pool parallelism you’ve specified. After setting the fork-join pool parallelism to 10, and increasing the number of elements in the collection to 50, I see the parallelism of the list-based stream rising only to 6, whereas the parallelism of the set-based stream never gets above 2.

Note, however, that this technique of submitting a task to a fork-join pool to run the parallel stream in that pool is an implementation “trick” and is not guaranteed to work. Indeed, the threads or thread pool that is used for execution of parallel streams is unspecified. By default, the common fork-join pool is used, but in different environments, different thread pools might end up being used. (Consider a container within an application server.)

In the java.util.stream.AbstractTask class, the LEAF_TARGET field determines the amount of splitting that is done, which in turn determines the amount of parallelism that can be achieved. The value of this field is based on ForkJoinPool.getCommonPoolParallelism() which of course uses the parallelism of the common pool, not whatever pool happens to be running the tasks.

Arguably this is a bug (see OpenJDK issue JDK-8190974), however, this entire area is unspecified anyway. However, this area of the system definitely needs development, for example in terms of splitting policy, the amount of parallelism available, dealing with blocking tasks, among other issues. A future release of the JDK may address some of these issues.

Meanwhile, it is possible to control the parallelism of the common fork-join pool through the use of system properties. If you add this line to your program,

System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "10");

and you run the streams in the common pool (or if you submit them to your own pool that has a sufficiently high level of parallelism set) you will observe that many more tasks are run in parallel.

You can also set this property on the command line using the -D option.

Again, this is not guaranteed behavior, and it may change in the future. But this technique will probably work for JDK 8 implementations for the forseeable future.

UPDATE 2019-06-12: The bug JDK-8190974 was fixed in JDK 10, and the fix has been backported to an upcoming JDK 8u release (8u222).

More Related Contents:

  1. Custom thread pool in Java 8 parallel stream
  2. Should I always use a parallel stream when possible?
  3. Why filter() after flatMap() is “not completely” lazy in Java streams?
  4. How can I throw CHECKED exceptions from inside Java 8 streams?
  5. Java 8 lambdas, Function.identity() or t->t
  6. Copy a stream to avoid “stream has already been operated upon or closed”
  7. Using Java 8’s Optional with Stream::flatMap
  8. Java 8 Streams: multiple filters vs. complex condition
  9. Java 8’s streams: why parallel stream is slower?
  10. Retrieving a List from a java.util.stream.Stream in Java 8
  11. Parallel streams, collectors and thread safety
  12. Filter Java Stream to 1 and only 1 element
  13. Java 8, Streams to find the duplicate elements
  14. Is this a bug in Files.lines(), or am I misunderstanding something about parallel streams?
  15. Why doesn’t java.util.Collection implement the new Stream interface?
  16. Find maximum, minimum, sum and average of a list in Java 8
  17. Does Java SE 8 have Pairs or Tuples?
  18. How to use a Java8 lambda to sort a stream in reverse order?
  19. Stream.skip behavior with unordered terminal operation
  20. Java 8 Collectors.toMap SortedMap
  21. Why parallel stream get collected sequentially in Java 8
  22. Java 8 grouping using custom collector?
  23. How can I throw CHECKED exceptions from inside Java 8 lambdas/streams?
  24. How to apply multiple predicates to a java.util.Stream?
  25. Why is Files.list() parallel stream performing so much slower than using Collection.parallelStream()?
  26. Non-interference exact meaning in Java 8 streams
  27. In Java 8 how do I transform a Map to another Map using a lambda?
  28. Convert String array to Map using Java 8 Lambda expressions
  29. Visualization of Java Stream parallelization
  30. Group sequences of values

Leave a Comment