selenium with scrapy for dynamic page

by

It really depends on how do you need to scrape the site and how and what data do you want to get.

Here’s an example how you can follow pagination on ebay using Scrapy+Selenium:

import scrapy
from selenium import webdriver

class ProductSpider(scrapy.Spider):
    name = "product_spider"
    allowed_domains = ['ebay.com']
    start_urls = ['http://www.ebay.com/sch/i.html?_odkw=books&_osacat=0&_trksid=p2045573.m570.l1313.TR0.TRC0.Xpython&_nkw=python&_sacat=0&_from=R40']

    def __init__(self):
        self.driver = webdriver.Firefox()

    def parse(self, response):
        self.driver.get(response.url)

        while True:
            next = self.driver.find_element_by_xpath('//td[@class="pagn-next"]/a')

            try:
                next.click()

                # get the data and write it to scrapy items
            except:
                break

        self.driver.close()

Here are some examples of “selenium spiders”:

There is also an alternative to having to use Selenium with Scrapy. In some cases, using ScrapyJS middleware is enough to handle the dynamic parts of a page. Sample real-world usage:

More Related Contents:

  1. how to scrape web page that is not written directly using HTML, but is auto-generated using JavaScript? [closed]
  2. How can I download a file on a click event using selenium?
  3. Python: find_element_by_css_selector
  4. ‘list’ object has no attribute ‘get_attribute’ while iterating through WebElements
  5. How to click a link by text with No Text in Python
  6. Selenium-Debugging: Element is not clickable at point (X,Y)
  7. Fetch all href link using selenium in python
  8. Python selenium error when trying to launch firefox
  9. Selenium – wait until element is present, visible and interactable
  10. How to select a drop-down menu value with Selenium using Python?
  11. WebDriverException: Message: ‘chromedriver’ executable needs to be in PATH while setting UserAgent through Selenium Chromedriver python
  12. Python : no module named selenium
  13. Of the many findElement(s)/By functions in Selenium, when would you use one over the other?
  14. How to simulate HTML5 Drag and Drop in Selenium Webdriver?
  15. How to fix Selenium WebDriverException: The browser appears to have exited before we could connect?
  16. Turning off logging in Selenium (from Python)
  17. SyntaxError: Failed to execute ‘evaluate’ on ‘Document’: The string ‘//img[contains(‘1236548597′)]’ is not a valid XPath expression
  18. “NoSuchWindowException: no such window: window was already closed” while switching tabs using Selenium and WebDriver through Python3
  19. Set chrome browser binary through chromedriver in Python
  20. Selenium on MAC, Message: ‘chromedriver’ executable may have wrong permissions
  21. Get all child elements
  22. How to get the scrapy failure URLs?
  23. unable to call firefox from selenium in python on AWS machine
  24. How to properly stop phantomjs execution
  25. Why switching to alert through selenium is not stable?
  26. IOError: [Errno 13] Permission denied: ‘geckodriver.log when running Python/Selenium
  27. How to fix python-selenium error “connection refused” when initializing a selenium webdriver?
  28. Webpage Is Detecting Selenium Webdriver with Chromedriver as a bot
  29. How to select multiple options from multi select list using Selenium-Python?
  30. Python Selenium driver.execute_script WebDriverException: Message: unknown error: call function result missing ‘value’

Leave a Comment