how to handle 302 redirect in scrapy

by

Forgot about middlewares in this scenario, this will do the trick:

meta = {'dont_redirect': True,'handle_httpstatus_list': [302]}

That said, you will need to include meta parameter when you yield your request:

yield Request(item['link'],meta = {
                  'dont_redirect': True,
                  'handle_httpstatus_list': [302]
              }, callback=self.your_callback)

More Related Contents:

  1. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  2. Scrapy Crawl URLs in Order
  3. Cannot install Lxml on Mac OS X 10.9
  4. ReactorNotRestartable error in while loop with scrapy
  5. How to run Scrapy from within a Python script
  6. Scrapy – Reactor not Restartable [duplicate]
  7. Scrapy Very Basic Example
  8. Scrapy and proxies
  9. Scrapy – how to manage cookies/sessions
  10. Scraping ajax pages using python
  11. How to integrate Flask & Scrapy?
  12. Crawling with an authenticated session in Scrapy
  13. Click a Button in Scrapy
  14. Access django models inside of Scrapy
  15. How can i use multiple requests and pass items in between them in scrapy python
  16. Using Scrapy with authenticated (logged in) user session
  17. Running Scrapy spiders in a Celery task
  18. scrapy: Call a function when a spider quits
  19. Scrapy crawl from script always blocks script execution after scraping
  20. crawl site that has infinite scrolling using python
  21. TypeError: Object of type ‘bytes’ is not JSON serializable
  22. getting Forbidden by robots.txt: scrapy
  23. Force my scrapy spider to stop crawling
  24. Get document DOCTYPE with BeautifulSoup
  25. How to use PyCharm to debug Scrapy projects
  26. Scrapy: ImportError: No module named items
  27. Locally run all of the spiders in Scrapy
  28. Creating a generic scrapy spider
  29. Scrapy CrawlSpider doesn’t crawl the first landing page
  30. find a word on a website and get its page link

Leave a Comment