Python parallel execution with selenium

by

Use joblib’s Parallel module to do that, its a great library for parallel execution.

Lets say we have a list of urls named urls and we want to take a screenshot of each one in parallel

First lets import the necessary libraries

from selenium import webdriver
from joblib import Parallel, delayed

Now lets define a function that takes a screenshot as base64

def take_screenshot(url):
    phantom = webdriver.PhantomJS('/path/to/phantomjs')
    phantom.get(url)
    screenshot = phantom.get_screenshot_as_base64()
    phantom.close()

    return screenshot

Now to execute that in parallel what you would do is

screenshots = Parallel(n_jobs=-1)(delayed(take_screenshot)(url) for url in urls)

When this line will finish executing, you will have in screenshots all of the data from all of the processes that ran.

Explanation about Parallel

  • Parallel(n_jobs=-1) means use all of the resources you can
  • delayed(function)(input) is joblib‘s way of creating the input for the function you are trying to run on parallel

More information can be found on the joblib docs

More Related Contents:

  1. How to sleep Selenium WebDriver in Python for milliseconds
  2. selenium.common.exceptions.WebDriverException: Message: ‘chromedriver’ executable needs to be in PATH error with Headless Chrome
  3. Wait until page is loaded with Selenium WebDriver for Python
  4. How to use Chrome Profile in Selenium Webdriver Python 3
  5. How to connect to Tor browser using Python
  6. How can I download a file on a click event using selenium?
  7. How to upload file ( picture ) with selenium, python
  8. InvalidArgumentException: Message: invalid argument: user data directory is already in use error using –user-data-dir to start Chrome using Selenium
  9. Log into gmail using Selenium in Python
  10. Select -> option abstraction
  11. Selenium not able to click on Get Data button on using Python
  12. selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities with Firefox 46 through Selenium
  13. Turning off logging in Selenium (from Python)
  14. Using the multiprocessing module for cluster computing
  15. urllib3.exceptions.ProtocolError: (‘Connection aborted.’, error(10054, ‘An existing connection was forcibly closed by the remote host’))
  16. How to hide Firefox window (Selenium WebDriver)?
  17. How to iterate over consecutive chunks of Pandas dataframe efficiently
  18. Python/Selenium incognito/private mode
  19. DeprecationWarning: use options instead of chrome_options error using ChromeDriver and Chrome through Selenium on Windows 10 system
  20. Selenium Python: How to web scrape the element text
  21. Open tor browser with selenium
  22. Python Selenium – get href value
  23. Selenium on MAC, Message: ‘chromedriver’ executable may have wrong permissions
  24. Get all child elements
  25. deciding among subprocess, multiprocessing, and thread in Python?
  26. Webdriver Exception:Process unexpectedly closed with status: 1
  27. Logic Operators in WebDriverWait Expected Conditions
  28. ElementNotInteractableException: Message: element not interactable error sending text in search field using Selenium Python
  29. Webdriver Screenshot
  30. Selenium + Python: How to stop page loading when certain element gets loaded?

Leave a Comment