Force my scrapy spider to stop crawling

by

In the latest version of Scrapy, available on GitHub, you can raise a CloseSpider exception to manually close a spider.

In the 0.14 release note doc is mentioned: “Added CloseSpider exception to manually close spiders (r2691)”

Example as per the docs:

def parse_page(self, response):
  if 'Bandwidth exceeded' in response.body:
    raise CloseSpider('bandwidth_exceeded')

See also: http://readthedocs.org/docs/scrapy/en/latest/topics/exceptions.html?highlight=closeSpider

More Related Contents:

  1. selenium with scrapy for dynamic page
  2. Scraping dynamic content using python-Scrapy
  3. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  4. Scrapy Crawl URLs in Order
  5. Cannot install Lxml on Mac OS X 10.9
  6. ReactorNotRestartable error in while loop with scrapy
  7. How to run Scrapy from within a Python script
  8. How to pass a user defined argument in scrapy spider
  9. “OSError: [Errno 1] Operation not permitted” when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)
  10. Scrapy and proxies
  11. Scrapy – how to manage cookies/sessions
  12. Scraping ajax pages using python
  13. passing selenium response url to scrapy
  14. Scrapy image download how to use custom filename
  15. Access django models inside of Scrapy
  16. How to get the scrapy failure URLs?
  17. how to filter duplicate requests based on url in scrapy
  18. How can I get all the plain text from a website with Scrapy?
  19. How can i use multiple requests and pass items in between them in scrapy python
  20. Using Scrapy with authenticated (logged in) user session
  21. Running Scrapy spiders in a Celery task
  22. scrapy: Call a function when a spider quits
  23. crawl site that has infinite scrolling using python
  24. How can I use different pipelines for different spiders in a single Scrapy project
  25. TypeError: Object of type ‘bytes’ is not JSON serializable
  26. Python Scrapy: Convert relative paths to absolute paths
  27. How to use PyCharm to debug Scrapy projects
  28. Scrapy: ImportError: No module named items
  29. Locally run all of the spiders in Scrapy
  30. Scrapy CrawlSpider doesn’t crawl the first landing page

Leave a Comment