how to filter duplicate requests based on url in scrapy

by

You can write custom middleware for duplicate removal and add it in settings

import os

from scrapy.dupefilter import RFPDupeFilter

class CustomFilter(RFPDupeFilter):
"""A dupe filter that considers specific ids in the url"""

    def __getid(self, url):
        mm = url.split("&refer")[0] #or something like that
        return mm

    def request_seen(self, request):
        fp = self.__getid(request.url)
        if fp in self.fingerprints:
            return True
        self.fingerprints.add(fp)
        if self.file:
            self.file.write(fp + os.linesep)

Then you need to set the correct DUPFILTER_CLASS in settings.py

DUPEFILTER_CLASS = 'scraper.duplicate_filter.CustomFilter'

It should work after that

More Related Contents:

  1. How to run Scrapy from within a Python script
  2. Scrapy – Reactor not Restartable [duplicate]
  3. How to pass a user defined argument in scrapy spider
  4. Click a Button in Scrapy
  5. crawl site that has infinite scrolling using python
  6. How can I use different pipelines for different spiders in a single Scrapy project
  7. getting Forbidden by robots.txt: scrapy
  8. Locally run all of the spiders in Scrapy
  9. Creating a generic scrapy spider
  10. Scrapy CrawlSpider doesn’t crawl the first landing page
  11. find a word on a website and get its page link
  12. selenium with scrapy for dynamic page
  13. Sending “User-agent” using Requests library in Python
  14. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  15. Scrapy Crawl URLs in Order
  16. Cannot install Lxml on Mac OS X 10.9
  17. ReactorNotRestartable error in while loop with scrapy
  18. “OSError: [Errno 1] Operation not permitted” when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)
  19. Anyone know of a good Python based web crawler that I could use?
  20. Scrapy and proxies
  21. Scrapy – how to manage cookies/sessions
  22. Scraping ajax pages using python
  23. Access django models inside of Scrapy
  24. How to get the scrapy failure URLs?
  25. How can i use multiple requests and pass items in between them in scrapy python
  26. Using Scrapy with authenticated (logged in) user session
  27. scrapy: Call a function when a spider quits
  28. Python: maximum recursion depth exceeded while calling a Python object
  29. Python Scrapy: Convert relative paths to absolute paths
  30. How can I scrape tooltips value from a Tableau graph embedded in a webpage

Leave a Comment