Scrapy CrawlSpider doesn’t crawl the first landing page

by

Just change your callback to parse_start_url and override it:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

class DownloadSpider(CrawlSpider):
    name="downloader"
    allowed_domains = ['bnt-chemicals.de']
    start_urls = [
        "http://www.bnt-chemicals.de",
    ]
    rules = (
        Rule(SgmlLinkExtractor(allow='prod'), callback='parse_start_url', follow=True),
    )
    fname = 0

    def parse_start_url(self, response):
        self.fname += 1
        fname="%s.txt" % self.fname

        with open(fname, 'w') as f:
            f.write('%s, %s\n' % (response.url, response.meta.get('depth', 0)))
            f.write('%s\n' % response.body)

More Related Contents:

  1. How to run Scrapy from within a Python script
  2. Scrapy – Reactor not Restartable [duplicate]
  3. How to pass a user defined argument in scrapy spider
  4. Click a Button in Scrapy
  5. how to filter duplicate requests based on url in scrapy
  6. crawl site that has infinite scrolling using python
  7. How can I use different pipelines for different spiders in a single Scrapy project
  8. getting Forbidden by robots.txt: scrapy
  9. Locally run all of the spiders in Scrapy
  10. Creating a generic scrapy spider
  11. find a word on a website and get its page link
  12. selenium with scrapy for dynamic page
  13. Sending “User-agent” using Requests library in Python
  14. Scraping dynamic content using python-Scrapy
  15. Scrapy Crawl URLs in Order
  16. Cannot install Lxml on Mac OS X 10.9
  17. “OSError: [Errno 1] Operation not permitted” when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)
  18. Scrapy and proxies
  19. passing selenium response url to scrapy
  20. Scrapy image download how to use custom filename
  21. Access django models inside of Scrapy
  22. How to get the scrapy failure URLs?
  23. How can I get all the plain text from a website with Scrapy?
  24. how to handle 302 redirect in scrapy
  25. How can i use multiple requests and pass items in between them in scrapy python
  26. scrapy: Call a function when a spider quits
  27. Python: maximum recursion depth exceeded while calling a Python object
  28. Python Scrapy: Convert relative paths to absolute paths
  29. How can I scrape tooltips value from a Tableau graph embedded in a webpage
  30. Scrapy: ImportError: No module named items

Leave a Comment