thread safety of MPI send using threads created with std::async

by

Thread-safety in MPI doesn’t work out of the box. First, you have to ensure that your implementation actually supports multiple threads making MPI calls at once. With some MPI implementations, for example Open MPI, this requires the library to be configured with special options at build time. Then you have to tell MPI to initialise at the appropriate thread support level. Currently the MPI standard defines four levels of thread support:

  • MPI_THREAD_SINGLE – means that the user code is single threaded. This is the default level at which MPI is initialised if MPI_Init() is used;
  • MPI_THREAD_FUNNELED – means that the user code is multithreaded, but only the main thread makes MPI calls. The main thread is the one which initialises the MPI library;
  • MPI_THREAD_SERIALIZED – means that the user code is multithreaded, but calls to the MPI library are serialised;
  • MPI_THREAD_MULTIPLE – means that the user code is multithreaded and all threads can make MPI calls at any time with no synchronisation whatsoever.

In order to initialise MPI with thread support, one has to use MPI_Init_thread() instead of MPI_Init():

int provided;

MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
if (provided < MPI_THREAD_MULTIPLE)
{
    printf("ERROR: The MPI library does not have full thread support\n");
    MPI_Abort(MPI_COMM_WORLD, 1);
}

Equivalent code with the obsoleted (and removed from MPI-3) C++ bindings:

int provided = MPI::Init_thread(argc, argv, MPI::THREAD_MULTIPLE);
if (provided < MPI::THREAD_MULTIPLE)
{
    printf("ERROR: The MPI library does not have full thread support\n");
    MPI::COMM_WORLD.Abort(1);
}

Thread support levels are ordered like this: MPI_THREAD_SINGLE < MPI_THREAD_FUNNELED < MPI_THREAD_SERIALIZED < MPI_THREAD_MULTIPLE, so any other provided level, different from MPI_THREAD_MULTIPLE would have lower numerical value – that’s why the if (...) code above is written so.

MPI_Init(&argc, &argv) is equivalent to MPI_Init_thread(&argc, &argv, MPI_THREAD_SINGLE, &provided). Implementations are not required to initialise exactly at the requested level – rather they could initialise at any other level (higher or lower), which is returned in the provided output argument.

For more information – see ยง12.4 of the MPI standard, freely available here.

With most MPI implementations, the thread support at level MPI_THREAD_SINGLE is actually equivalent to that provided at level MPI_THREAD_SERIALIZED – exactly what you observe in your case.

Since you’ve not specified which MPI implementation you use, here comes a handy list.

I’ve already said that Open MPI has to be compiled with the proper flags enabled in order to support MPI_THREAD_MULTIPLE. But there is another catch – its InfiniBand component is not thread-safe and hence Open MPI would not use native InfiniBand communication when initialised at full thread support level.

Intel MPI comes in two different flavours – one with and one without support for full multithreading. Multithreaded support is enabled by passing the -mt_mpi option to the MPI compiler wrapper which enables linking with the MT version. This option is also implied if OpenMP support or the autoparalleliser is enabled. I am not aware how the InfiniBand driver in IMPI works when full thread support is enabled.

MPICH(2) does not support InfiniBand, hence it is thread-safe and probably most recent versions provide MPI_THREAD_MULTIPLE support out of the box.

MVAPICH is the basis on which Intel MPI is built and it supports InfiniBand. I have no idea how it behaves at full thread support level when used on a machine with InfiniBand.

The note about multithreaded InfiniBand support is important since lot of compute clusters nowadays use InfiniBand fabrics. With the IB component (openib BTL in Open MPI) disabled, most MPI implementations switch to another protocol, for example TCP/IP (tcp BTL in Open MPI), which results in much slower and more latent communication.

More Related Contents:

  1. C++11 introduced a standardized memory model. What does it mean? And how is it going to affect C++ programming?
  2. What happens to a detached thread when main() exits?
  3. Multithreading program stuck in optimized mode but runs normally in -O0
  4. Thread pooling in C++11
  5. What are the correct link options to use std::thread in GCC under linux?
  6. Why is the destructor of a future returned from `std::async` blocking?
  7. What does the thread_local mean in C++11?
  8. How to check if a std::thread is still running?
  9. Does MinGW-w64 support std::thread out of the box when using the Win32 threading model?
  10. C++11 thread-safe queue
  11. What exactly is std::atomic?
  12. How to implement multithread safe singleton in C++11 without using
  13. Difference between pointer and reference as thread parameter
  14. std::unique_lock or std::lock_guard?
  15. What do each memory_order mean?
  16. Does the C++ volatile keyword introduce a memory fence?
  17. Which std::async implementations use thread pools?
  18. How can I use something like std::vector?
  19. Get the status of a std::future
  20. Delayed start of a thread in C++ 11
  21. Thread safety of std::map for read-only operations
  22. Is it smart to replace boost::thread and boost::mutex with c++11 equivalents?
  23. C++11 STL containers and thread safety
  24. Can I use std::async without waiting for the future limitation?
  25. C++ terminate called without an active exception
  26. Understanding std::atomic::compare_exchange_weak() in C++11
  27. Double-Checked Lock Singleton in C++11
  28. How to implement a unmanaged thread-safe collection when I get this error: is not supported when compiling with /clr
  29. C++11 Implementation of Spinlock using header “
  30. About thread-safety of weak_ptr

Leave a Comment