passing selenium response url to scrapy

by

Use Downloader Middleware to catch selenium-required pages before you process them regularly with Scrapy:

The downloader middleware is a framework of hooks into Scrapy’s request/response processing. It’s a light, low-level system for globally altering Scrapy’s requests and responses.

Here’s a very basic example using PhantomJS:

from scrapy.http import HtmlResponse
from selenium import webdriver

class JSMiddleware(object):
    def process_request(self, request, spider):
        driver = webdriver.PhantomJS()
        driver.get(request.url)

        body = driver.page_source
        return HtmlResponse(driver.current_url, body=body, encoding='utf-8', request=request)

Once you return that HtmlResponse (or a TextResponse if that’s what you really want), Scrapy will cease processing downloaders and drop into the spider’s parse method:

If it returns a Response object, Scrapy won’t bother calling any other
process_request() or process_exception() methods, or the appropriate
download function; it’ll return that response. The process_response()
methods of installed middleware is always called on every response.

In this case, you can continue to use your spider’s parse method as you normally would with HTML, except that the JS on the page has already been executed.

Tip: Since the Downloader Middleware’s process_request method accepts the spider as an argument, you can add a conditional in the spider to check whether you need to process JS at all, and that will let you handle both JS and non-JS pages with the exact same spider class.

More Related Contents:

  1. selenium with scrapy for dynamic page
  2. crawl site that has infinite scrolling using python
  3. ChromeDriver Error: ‘chromedriver’ executable needs to be in PATH [duplicate]
  4. How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
  5. How to get attribute of element from Selenium?
  6. How to work with a specific version of ChromeDriver while Chrome Browser gets updated automatically through Python selenium
  7. selenium.common.exceptions.InvalidSelectorException with “span:contains(‘string’)”
  8. find_element_by_* commands are deprecated in selenium
  9. Unable to use Selenium to automate Chase site login
  10. Selenium Element not visible exception
  11. Scrapy – how to manage cookies/sessions
  12. How to disable CSS in Python selenium using ChromeOptions
  13. Dynamic dropdown doesn’t populate with auto suggestions on https://www.nseindia.com/ when values are passed using Selenium and Python
  14. ElementNotVisibleException : Selenium Python
  15. How to open a link embeded in a webelement with in the main tab, in a new tab of the same window using Control + Click of Selenium Webdriver
  16. How can Geckodriver/Firefox work without Marionette? (running python selenium 3 against FF 53)
  17. How to ignore zoom setting
  18. Python: use cookie to login with Selenium
  19. Selenium using too much RAM with Firefox
  20. How to access to ‘rect’ type element through Selenium-Python
  21. Using Scrapy with authenticated (logged in) user session
  22. How can we use Selenium Webdriver in colab.research.google.com?
  23. How to use Brave web browser with python, selenium and chromedriver?
  24. Selenium – visibility_of_element_located: __init__() takes exactly 2 arguments (3 given)
  25. How to install Chrome Extension using Selenium & Python
  26. Scrapy: ImportError: No module named items
  27. Browser performance tests through selenium
  28. Is it possible to run multiple instances of one selenium test at once?
  29. How to click on Load More button within Google Trends and print all the titles through Selenium and Python
  30. Selenium / Python – Selecting via css selector

Leave a Comment