how do we choose –nthreads and –nprocs per worker in dask distributed?

by

It depends on your workload

By default Dask creates a single process with as many threads as you have logical cores on your machine (as determined by multiprocessing.cpu_count()). 

dask-worker ... --nprocs 1 --nthreads 8  # assuming you have eight cores
dask-worker ...                          # this is actually the default setting

Using few processes and many threads per process is good if you are doing mostly numeric workloads, such as are common in Numpy, Pandas, and Scikit-Learn code, which is not affected by Python’s Global Interpreter Lock (GIL).

However, if you are spending most of your compute time manipulating Pure Python objects like strings or dictionaries then you may want to avoid GIL issues by having more processes with fewer threads each

dask-worker ... --nprocs 8 --nthreads 1

Based on benchmarking you may find that a more balanced split is better

dask-worker ... --nprocs 4 --nthreads 2

Using more processes avoids GIL issues, but adds costs due to inter-process communication. You would want to avoid many processes if your computations require a lot of inter-worker communication..

More Related Contents:

  1. Why isn’t RDBMS Partition Tolerant in CAP Theorem and why is it Available?
  2. Spark – repartition() vs coalesce()
  3. Make Pandas DataFrame apply() use all cores?
  4. Flattening Rows in Spark
  5. Meaning of inter_op_parallelism_threads and intra_op_parallelism_threads
  6. What determines Kafka consumer offset?
  7. What are workers, executors, cores in Spark Standalone cluster?
  8. C++ Winsock P2P
  9. How a server can make a session with a client in RMI
  10. What is a task in Spark? How does the Spark worker execute the jar file?
  11. Why shared libraries between microservices are bad? [closed]
  12. Scatter Matrix Blocks of Different Sizes using MPI
  13. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  14. How does Spark aggregate function – aggregateByKey work?
  15. What is the difference between cache and persist?
  16. Easiest way to install Python dependencies on Spark executor nodes?
  17. python dask DataFrame, support for (trivially parallelizable) row apply?
  18. Exploding nested Struct in Spark dataframe
  19. Change File Split size in Hadoop
  20. What is spark.driver.maxResultSize?
  21. Python Multiprocessing with Distributed Cluster
  22. how to parallelize many (fuzzy) string comparisons using apply in Pandas?
  23. dask dataframe how to convert column to to_datetime

Leave a Comment