Scrapy Crawl URLs in Order

by

Scrapy Request has a priority attribute now.

If you have many Request in a function and want to process a particular request first, you can set:

def parse(self, response):
    url="http://www.example.com/first"
    yield Request(url=url, callback=self.parse_data, priority=1)

    url="http://www.example.com/second"
    yield Request(url=url, callback=self.parse_data)

Scrapy will process the one with priority=1 first.

More Related Contents:

  1. How do I sort a dictionary by value?
  2. How to sort a dataFrame in python pandas by two or more columns?
  3. how to sort pandas dataframe from one column
  4. Python analog of PHP’s natsort function (sort a list using a “natural order” algorithm) [duplicate]
  5. Rank items in an array using Python/NumPy, without sorting array twice
  6. Sort list of lists ascending and then descending
  7. Custom Python list sorting
  8. Scrapy – Reactor not Restartable [duplicate]
  9. heapq with custom compare predicate
  10. How can I “zip sort” parallel numpy arrays?
  11. Crawling with an authenticated session in Scrapy
  12. Python Math – TypeError: ‘NoneType’ object is not subscriptable
  13. What is the complexity of the sorted() function?
  14. Why is the cmp parameter removed from sort/sorted in Python3.0?
  15. Sort filenames in directory in ascending order [duplicate]
  16. Making a python user-defined class sortable, hashable
  17. Sort invariant for numpy.argsort with multiple dimensions
  18. Given parallel lists, how can I sort one while permuting (rearranging) the other in the same way?
  19. Pandas DataFrame sort by categorical column but by specific class ordering
  20. Scrapy crawl from script always blocks script execution after scraping
  21. crawl site that has infinite scrolling using python
  22. getting Forbidden by robots.txt: scrapy
  23. Sort a list of tuples by second value, reverse=True and then by key, reverse=False
  24. Python sorting complexity on sorted list
  25. Can I sort text by its numeric value in Python?
  26. Get document DOCTYPE with BeautifulSoup
  27. Retrieve the two highest item from a list containing 100,000 integers
  28. How to sort a list of dictionaries by a value of the dictionary in Python?
  29. Scrapy CrawlSpider doesn’t crawl the first landing page
  30. What algorithm does python’s sorted() use? [duplicate]

Leave a Comment