MPI: blocking vs non-blocking

by

Blocking communication is done using MPI_Send() and MPI_Recv(). These functions do not return (i.e., they block) until the communication is finished. Simplifying somewhat, this means that the buffer passed to MPI_Send() can be reused, either because MPI saved it somewhere, or because it has been received by the destination. Similarly, MPI_Recv() returns when the receive buffer has been filled with valid data.

In contrast, non-blocking communication is done using MPI_Isend() and MPI_Irecv(). These function return immediately (i.e., they do not block) even if the communication is not finished yet. You must call MPI_Wait() or MPI_Test() to see whether the communication has finished.

Blocking communication is used when it is sufficient, since it is somewhat easier to use. Non-blocking communication is used when necessary, for example, you may call MPI_Isend(), do some computations, then do MPI_Wait(). This allows computations and communication to overlap, which generally leads to improved performance.

Note that collective communication (e.g., all-reduce) is only available in its blocking version up to MPIv2. IIRC, MPIv3 introduces non-blocking collective communication.

A quick overview of MPI’s send modes can be seen here.

More Related Contents:

  1. sending blocks of 2D array in C using MPI
  2. Optimal number of threads per core
  3. OpenMP program is slower than sequential one
  4. Julia: How to copy data to another processor in Julia
  5. Processing vec in parallel: how to do safely, or without using unstable features?
  6. Scatter Matrix Blocks of Different Sizes using MPI
  7. MPI Reading from a text file
  8. GNU parallel not working at all
  9. What is the purpose of the “PAUSE” instruction in x86?
  10. Parallel Algorithms for Generating Prime Numbers (possibly using Hadoop’s map reduce)
  11. MPI_Type_create_subarray and MPI_Gather
  12. multiple targets from one recipe and parallel execution
  13. Is there a good way to extract chunks of data from a java 8 stream?
  14. What is the difference between concurrency and parallelism?
  15. What are the differences between the threading and multiprocessing modules?
  16. How to articulate the difference between asynchronous and parallel programming?
  17. Can I use the task parallel library in a .Net 3.5 project?
  18. Scaling solutions for MySQL (Replication, Clustering)
  19. Run several jobs parallelly and Efficiently
  20. data.table and parallel computing
  21. Parallel Processing in python
  22. Set hadoop system user for client embedded in Java webapp
  23. Cuda kernel returning vectors
  24. Understanding concurrent file writes from multiple processes
  25. Modify object in python multiprocessing
  26. What determines the number of threads a Java ForkJoinPool creates?
  27. OpenMP: What is the benefit of nesting parallelizations?
  28. Tensorflow and Multiprocessing: Passing Sessions
  29. Different execution policies at runtime
  30. MPI hangs on MPI_Send for large messages

Leave a Comment