Is there a good way to extract chunks of data from a java 8 stream?

My approach to bulk operations with chunking is to use a partitioning spliterator wrapper, and another wrapper which overrides the default splitting policy (arithmetic progression of batch sizes in increments of 1024) to simple fixed-batch splitting. Use it like this: Stream<OriginalType> existingStream = …; Stream<List<OriginalType>> partitioned = partition(existingStream, 100, 1); partitioned.forEach(chunk -> … process the … Read more

Processing vec in parallel: how to do safely, or without using unstable features?

Today the rayon crate is the de facto standard for this sort of thing: use rayon::prelude::*; fn main() { let mut data = vec![1, 2, 3]; data.par_iter_mut() .enumerate() .for_each(|(i, x)| *x = 10 + i as u32); assert_eq!(vec![10, 11, 12], data); } Note that this is just one line different from the single-threaded version using … Read more

MPI: blocking vs non-blocking

Blocking communication is done using MPI_Send() and MPI_Recv(). These functions do not return (i.e., they block) until the communication is finished. Simplifying somewhat, this means that the buffer passed to MPI_Send() can be reused, either because MPI saved it somewhere, or because it has been received by the destination. Similarly, MPI_Recv() returns when the receive … Read more