How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?

by

I am looking for a way to limit a python scripts CPU usage (not priority but the number of CPU cores) with python code.

Run you application with taskset or numactl.

For example, to make your application utilize only the first 4 CPUs do:

taskset --cpu-list 0-3 <app>

These tools, however, limit the process to use specific CPUs, not the total number of used CPUs. For best results they require those CPUs to be isolated from the OS process scheduler, so that the scheduler doesn’t run any other processes on those CPUs. Otherwise, if the specified CPUs are currently running other threads, while other CPUs are idle, your threads won’t be able to run on other idle CPUs and will have to queue up for these specific CPUs, which isn’t ideal.

Using cgroups you can limit your processes/threads to use a specific fraction of available CPU resources without limiting to specific CPUs, but cgroups setup is less trivial.

More Related Contents:

  1. multiprocessing: Understanding logic behind `chunksize`
  2. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  3. Multiprocessing AsyncResult.get() hangs in Python 3.7.2 but not in 3.6
  4. How to use multiprocessing pool.map with multiple arguments
  5. Python multiprocessing PicklingError: Can’t pickle
  6. Sharing a result queue among several processes
  7. Shared variable in python’s multiprocessing
  8. multiprocessing.Pool: What’s the difference between map_async and imap?
  9. Script using multiprocessing module does not terminate
  10. multiprocessing vs multithreading vs asyncio in Python 3
  11. Concurrent.futures vs Multiprocessing in Python 3
  12. How to prevent BrokenPipeError when doing a flush in Python?
  13. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  14. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  15. Python multiprocessing.Pool: AttributeError
  16. How to get the return value of a function passed to multiprocessing.Process?
  17. Python selenium multiprocessing
  18. How to share numpy random state of a parent process with child processes?
  19. multiprocessing vs multithreading vs asyncio
  20. Python OpenCV multiprocessing cv2.VideoCapture mp4
  21. how to to terminate process using python’s multiprocessing
  22. multiprocessing pool example does not work and freeze the kernel
  23. Using 100% of all cores with the multiprocessing module
  24. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  25. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  26. Python multiprocessing application is getting stuck in docker container
  27. pyvenv-3.4 returned non-zero exit status 1
  28. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  29. How can I abort a task in a multiprocessing.Pool after a timeout?
  30. How can I share a class between processes?

Leave a Comment