How to run Scrapy from within a Python script

by

All other answers reference Scrapy v0.x. According to the updated docs, Scrapy 1.0 demands:

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
    # Your spider definition
    ...

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished

More Related Contents:

  1. Click a Button in Scrapy
  2. find a word on a website and get its page link
  3. selenium with scrapy for dynamic page
  4. Scraping dynamic content using python-Scrapy
  5. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  6. Scrapy – Reactor not Restartable [duplicate]
  7. How to pass a user defined argument in scrapy spider
  8. Scrapy Very Basic Example
  9. Scraping ajax pages using python
  10. How to get the scrapy failure URLs?
  11. how to filter duplicate requests based on url in scrapy
  12. How can I get all the plain text from a website with Scrapy?
  13. crawl site that has infinite scrolling using python
  14. How can I use different pipelines for different spiders in a single Scrapy project
  15. getting Forbidden by robots.txt: scrapy
  16. How can I scrape tooltips value from a Tableau graph embedded in a webpage
  17. Python: Disable images in Selenium Google ChromeDriver
  18. Locally run all of the spiders in Scrapy
  19. Creating a generic scrapy spider
  20. Scrapy CrawlSpider doesn’t crawl the first landing page
  21. How to find elements by class
  22. Crawling with an authenticated session in Scrapy
  23. Scraping dynamic content in a website [duplicate]
  24. python: [Errno 10054] An existing connection was forcibly closed by the remote host
  25. Scrapy crawl from script always blocks script execution after scraping
  26. BeautifulSoup: Get the contents of a specific table
  27. Get document DOCTYPE with BeautifulSoup
  28. Scrapy: ImportError: No module named items
  29. Python 3: using requests does not get the full content of a web page
  30. post request using python to asp.net page

Leave a Comment