How to give URL to scrapy for crawling?

by

I’m not really sure about the commandline option. However, you could write your spider like this.

class MySpider(BaseSpider):

    name="my_spider"    

    def __init__(self, *args, **kwargs): 
      super(MySpider, self).__init__(*args, **kwargs) 

      self.start_urls = [kwargs.get('start_url')]

And start it like:
scrapy crawl my_spider -a start_url="http://some_url"

More Related Contents:

  1. How to run Scrapy from within a Python script
  2. Scrapy – Reactor not Restartable [duplicate]
  3. How to pass a user defined argument in scrapy spider
  4. Run a Scrapy spider in a Celery Task
  5. Click a Button in Scrapy
  6. how to filter duplicate requests based on url in scrapy
  7. crawl site that has infinite scrolling using python
  8. How can I use different pipelines for different spiders in a single Scrapy project
  9. getting Forbidden by robots.txt: scrapy
  10. scrapy- how to stop Redirect (302)
  11. Locally run all of the spiders in Scrapy
  12. Creating a generic scrapy spider
  13. Scrapy CrawlSpider doesn’t crawl the first landing page
  14. find a word on a website and get its page link
  15. Writing a web crawler in Java [closed]
  16. selenium with scrapy for dynamic page
  17. Headless Browser and scraping – solutions [closed]
  18. Scrapy Crawl URLs in Order
  19. Cannot install Lxml on Mac OS X 10.9
  20. “OSError: [Errno 1] Operation not permitted” when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)
  21. Scrapy and proxies
  22. Fetch contents(loaded through AJAX call) of a web page
  23. Access django models inside of Scrapy
  24. How to find all links / pages on a website
  25. Nodejs: Async request with a list of URL
  26. Python: maximum recursion depth exceeded while calling a Python object
  27. Submit form with no submit button in rvest
  28. selenium implicitly wait doesn’t work
  29. How to write a crawler?
  30. Crawling the Google Play store

Leave a Comment