Python selenium multiprocessing

by

how can I reduce the execution time using selenium when it is made to run using multiprocessing

A lot of time in your solution is spent on launching the webdriver for each URL. You can reduce this time by launching the driver only once per thread:

(... skipped for brevity ...)

threadLocal = threading.local()

def get_driver():
  driver = getattr(threadLocal, 'driver', None)
  if driver is None:
    chromeOptions = webdriver.ChromeOptions()
    chromeOptions.add_argument("--headless")
    driver = webdriver.Chrome(chrome_options=chromeOptions)
    setattr(threadLocal, 'driver', driver)
  return driver


def get_title(url):
  driver = get_driver()
  driver.get(url)
  (...)

(...)

On my system this reduces the time from 1m7s to just 24.895s, a ~35% improvement. To test yourself, download the full script.

Note: ThreadPool uses threads, which are constrained by the Python GIL. That’s ok if for the most part the task is I/O bound. Depending on the post-processing you do with the scraped results, you may want to use a multiprocessing.Pool instead. This launches parallel processes which as a group are not constrained by the GIL. The rest of the code stays the same.

More Related Contents:

  1. how to scrape web page that is not written directly using HTML, but is auto-generated using JavaScript? [closed]
  2. WebDriverWait not working as expected
  3. How to make Firefox headless programmatically in Selenium with Python?
  4. multiprocessing: Understanding logic behind `chunksize`
  5. How to deal with certificates using Selenium?
  6. Scraping Google Finance (BeautifulSoup)
  7. Scrolling to element using webdriver?
  8. Python: find_element_by_css_selector
  9. How do you fix the “element not interactable” exception?
  10. ‘list’ object has no attribute ‘get_attribute’ while iterating through WebElements
  11. Scrape website with dynamic mouseover event
  12. Dynamic dropdown doesn’t populate with auto suggestions on https://www.nseindia.com/ when values are passed using Selenium and Python
  13. Concurrent.futures vs Multiprocessing in Python 3
  14. Selenium gives “selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary” on Mac
  15. selenium – Failed to execute ‘evaluate’ on ‘Document’: The string is not a valid XPath expression
  16. How to share numpy random state of a parent process with child processes?
  17. ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
  18. Scraping dynamic data selenium – Unable to locate element
  19. multiprocessing vs multithreading vs asyncio
  20. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  21. Python OpenCV multiprocessing cv2.VideoCapture mp4
  22. Using 100% of all cores with the multiprocessing module
  23. Python selenium error when trying to launch firefox
  24. Handle notifications in Python + Selenium Chrome WebDriver
  25. “Failed to decode response from marionette” message in Python/Firefox headless scraping script
  26. Scraping Data from a website which uses Power BI – retrieving data from Power BI on a website
  27. Can’t use chrome driver for Selenium
  28. find a word on a website and get its page link
  29. Capture AJAX response with selenium and python
  30. Multiprocessing Share Unserializable Objects Between Processes

Leave a Comment