The twisted reactor cannot be restarted. A work around for this is to let the celery task fork a new child process for each crawl you want to execute as proposed in the following post: Running Scrapy spiders in a Celery task This gets around the “reactor cannot be restart-able” issue by utilizing the multiprocessing … Read more
By default, CrawlerProcess‘s .start() will stop the Twisted reactor it creates when all crawlers have finished. You should call process.start(stop_after_crawl=False) if you create process in each iteration. Another option is to handle the Twisted reactor yourself and use CrawlerRunner. The docs have an example on doing that.
try running these commands xcode-select –install xcode-select –switch /Applications/Xcode.app xcode-select –switch /Library/Developer/CommandLineTools just to make sure your setup is properly applied