ReactorNotRestartable error in while loop with scrapy

by

By default, CrawlerProcess‘s .start() will stop the Twisted reactor it creates when all crawlers have finished.

You should call process.start(stop_after_crawl=False) if you create process in each iteration.

Another option is to handle the Twisted reactor yourself and use CrawlerRunner. The docs have an example on doing that.

More Related Contents:

  1. “OSError: [Errno 1] Operation not permitted” when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)
  2. Scrapy crawl from script always blocks script execution after scraping
  3. How to use PyCharm to debug Scrapy projects
  4. python 2.7 script to add a column
  5. Python 2: Conditional statement behaviour of lists with greater/smaller [duplicate]
  6. Zip lists in Python
  7. Insert an element at a specific index in a list and return the updated list
  8. Cannot install Lxml on Mac OS X 10.9
  9. Backporting Python 3 open(encoding=”utf-8″) to Python 2
  10. How to pip or easy_install tkinter on Windows
  11. pip install gives error: Unable to find vcvarsall.bat
  12. Python dynamic function creation with custom names
  13. gcc-4.2 failed with exit status 1
  14. What does an ‘r’ represent before a string in python? [duplicate]
  15. Sort a list to form the largest possible number
  16. Get unicode code point of a character using Python
  17. How to split dictionary into multiple dictionaries fast
  18. Distributed lock manager for Python
  19. Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str
  20. How can I copy a Python string?
  21. Connection Timeout with Elasticsearch
  22. How to locate and insert a value in a text box (input) using Python Selenium?
  23. Multiple keys per value
  24. Python packages not installing in virtualenv using pip
  25. flatten list of list through list comprehension
  26. Why do I get “‘str’ object has no attribute ‘read'” when trying to use `json.load` on a string? [duplicate]
  27. Catching exception in context manager __enter__()
  28. confused about random_state in decision tree of scikit learn
  29. Is possible to mapping view with class using mapper in SqlAlchemy?
  30. How to handle IncompleteRead: in python

Leave a Comment