Anyone know of a good Python based web crawler that I could use?

by
  • Mechanize is my favorite; great high-level browsing capabilities (super-simple form filling and submission).
  • Twill is a simple scripting language built on top of Mechanize
  • BeautifulSoup + urllib2 also works quite nicely.
  • Scrapy looks like an extremely promising project; it’s new.

More Related Contents:

  1. Python Google Web Crawler
  2. Sending “User-agent” using Requests library in Python
  3. TypeError: can’t use a string pattern on a bytes-like object in re.findall()
  4. How to run Scrapy from within a Python script
  5. Scrapy – Reactor not Restartable [duplicate]
  6. How to pass a user defined argument in scrapy spider
  7. Click a Button in Scrapy
  8. how to filter duplicate requests based on url in scrapy
  9. python: [Errno 10054] An existing connection was forcibly closed by the remote host
  10. crawl site that has infinite scrolling using python
  11. How can I use different pipelines for different spiders in a single Scrapy project
  12. getting Forbidden by robots.txt: scrapy
  13. Python: maximum recursion depth exceeded while calling a Python object
  14. How can I scrape tooltips value from a Tableau graph embedded in a webpage
  15. Python: Disable images in Selenium Google ChromeDriver
  16. Locally run all of the spiders in Scrapy
  17. Creating a generic scrapy spider
  18. Scrapy CrawlSpider doesn’t crawl the first landing page
  19. find a word on a website and get its page link
  20. Passing Variables to Subprocess.Popen
  21. Why does this code for initializing a list of lists apparently link the lists together? [duplicate]
  22. Python and pip, list all versions of a package that’s available?
  23. How can I search sub-folders using glob.glob module? [duplicate]
  24. Why `torch.cuda.is_available()` returns False even after installing pytorch with cuda?
  25. ImportError: cannot import name NUMPY_MKL
  26. How does Python’s “super” do the right thing?
  27. PIP: “Cannot uninstall ‘ipython’. It is a distutils installed project and thus we cannot accurately determine…” [duplicate]
  28. Django circular model reference
  29. Heroku fails to install pywin32 library
  30. PyCharm hangs on ‘scanning files to index’ background task

Leave a Comment