How to pass a user defined argument in scrapy spider

by

Spider arguments are passed in the crawl command using the -a option. For example:

scrapy crawl myspider -a category=electronics -a domain=system

Spiders can access arguments as attributes:

class MySpider(scrapy.Spider):
    name="myspider"

    def __init__(self, category='', **kwargs):
        self.start_urls = [f'http://www.example.com/{category}']  # py36
        super().__init__(**kwargs)  # python3

    def parse(self, response)
        self.log(self.domain)  # system

Taken from the Scrapy doc: http://doc.scrapy.org/en/latest/topics/spiders.html#spider-arguments

Update 2013: Add second argument

Update 2015: Adjust wording

Update 2016: Use newer base class and add super, thanks @Birla

Update 2017: Use Python3 super

# previously
super(MySpider, self).__init__(**kwargs)  # python2

Update 2018: As @eLRuLL points out, spiders can access arguments as attributes

More Related Contents:

  1. How to run Scrapy from within a Python script
  2. Scrapy – Reactor not Restartable [duplicate]
  3. Click a Button in Scrapy
  4. how to filter duplicate requests based on url in scrapy
  5. crawl site that has infinite scrolling using python
  6. How can I use different pipelines for different spiders in a single Scrapy project
  7. getting Forbidden by robots.txt: scrapy
  8. Locally run all of the spiders in Scrapy
  9. Creating a generic scrapy spider
  10. Scrapy CrawlSpider doesn’t crawl the first landing page
  11. find a word on a website and get its page link
  12. selenium with scrapy for dynamic page
  13. Sending “User-agent” using Requests library in Python
  14. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  15. Scrapy Crawl URLs in Order
  16. Cannot install Lxml on Mac OS X 10.9
  17. ReactorNotRestartable error in while loop with scrapy
  18. “OSError: [Errno 1] Operation not permitted” when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)
  19. Anyone know of a good Python based web crawler that I could use?
  20. Scrapy and proxies
  21. passing selenium response url to scrapy
  22. Scrapy image download how to use custom filename
  23. Access django models inside of Scrapy
  24. How to get the scrapy failure URLs?
  25. How can i use multiple requests and pass items in between them in scrapy python
  26. Using Scrapy with authenticated (logged in) user session
  27. scrapy: Call a function when a spider quits
  28. Python: maximum recursion depth exceeded while calling a Python object
  29. Python Scrapy: Convert relative paths to absolute paths
  30. How can I scrape tooltips value from a Tableau graph embedded in a webpage

Leave a Comment