Creating a generic scrapy spider

by

You could create a run-time spider which is evaluated by the interpreter. This code piece could be evaluated at runtime like so:

a = open("test.py")
from compiler import compile
d = compile(a.read(), 'spider.py', 'exec')
eval(d)

MySpider
<class '__main__.MySpider'>
print MySpider.start_urls
['http://www.somedomain.com']

More Related Contents:

  1. How to run Scrapy from within a Python script
  2. Scrapy – Reactor not Restartable [duplicate]
  3. How to pass a user defined argument in scrapy spider
  4. Click a Button in Scrapy
  5. how to filter duplicate requests based on url in scrapy
  6. crawl site that has infinite scrolling using python
  7. How can I use different pipelines for different spiders in a single Scrapy project
  8. getting Forbidden by robots.txt: scrapy
  9. Locally run all of the spiders in Scrapy
  10. Scrapy CrawlSpider doesn’t crawl the first landing page
  11. find a word on a website and get its page link
  12. Python Google Web Crawler
  13. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  14. TypeError: can’t use a string pattern on a bytes-like object in re.findall()
  15. ReactorNotRestartable error in while loop with scrapy
  16. Scrapy Very Basic Example
  17. Anyone know of a good Python based web crawler that I could use?
  18. Scrapy – how to manage cookies/sessions
  19. Scraping ajax pages using python
  20. How to integrate Flask & Scrapy?
  21. Crawling with an authenticated session in Scrapy
  22. how to handle 302 redirect in scrapy
  23. python: [Errno 10054] An existing connection was forcibly closed by the remote host
  24. Running Scrapy spiders in a Celery task
  25. Scrapy crawl from script always blocks script execution after scraping
  26. TypeError: Object of type ‘bytes’ is not JSON serializable
  27. Force my scrapy spider to stop crawling
  28. Get document DOCTYPE with BeautifulSoup
  29. How to use PyCharm to debug Scrapy projects
  30. Python: Disable images in Selenium Google ChromeDriver

Leave a Comment