How to use PyCharm to debug Scrapy projects

by

The scrapy command is a python script which means you can start it from inside PyCharm.

When you examine the scrapy binary (which scrapy) you will notice that this is actually a python script:

#!/usr/bin/python

from scrapy.cmdline import execute
execute()

This means that a command like
scrapy crawl IcecatCrawler can also be executed like this: python /Library/Python/2.7/site-packages/scrapy/cmdline.py crawl IcecatCrawler

Try to find the scrapy.cmdline package.
In my case the location was here: /Library/Python/2.7/site-packages/scrapy/cmdline.py

Create a run/debug configuration inside PyCharm with that script as script. Fill the script parameters with the scrapy command and spider. In this case crawl IcecatCrawler.

Like this:
PyCharm Run/Debug Configuration

Put your breakpoints anywhere in your crawling code and it should work™.

More Related Contents:

  1. Add pygame module in PyCharm IDE
  2. ReactorNotRestartable error in while loop with scrapy
  3. “OSError: [Errno 1] Operation not permitted” when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)
  4. How to configure Python Kivy for PyCharm on Windows?
  5. Symbol not found: __PyCodecInfo_GetIncrementalDecoder
  6. Kivy error (python 2.7) : sdl2 import error
  7. Pycharm environment different than command line
  8. pydev breakpoints not working
  9. Python debugging tips [closed]
  10. How to sort a dataFrame in python pandas by two or more columns?
  11. How to remove \xa0 from string in Python?
  12. How can I selectively escape percent (%) in Python strings?
  13. Why isn’t my Pandas ‘apply’ function referencing multiple columns working? [closed]
  14. Python MySQLdb TypeError: not all arguments converted during string formatting
  15. Split a python list into other “sublists” i.e smaller lists [duplicate]
  16. Src layout to dispense .src prefix in imports? Activate venv in PyCharm terminal for development installs
  17. What is the right way to debug in iPython notebook?
  18. Insert a Pandas Dataframe into mongodb using PyMongo
  19. How can I open two consoles from a single script
  20. Click a Button in Scrapy
  21. How to specify where a Tkinter window opens?
  22. Python 2.7: Print to File
  23. Basemap import error in PyCharm — KeyError: ‘PROJ_LIB’
  24. Python to automatically select serial ports (for Arduino)
  25. PyCharm and PYTHONPATH
  26. Python how to replace backslash with re.sub()
  27. Why does updating one dictionary object affect other?
  28. ModuleNotFoundError: No module named ‘requests’ after pip install [duplicate]
  29. Python 2.7 : LookupError: unknown encoding: cp65001 [duplicate]
  30. Pycharm: run only part of my Python file

Leave a Comment