Can scrapy be used to scrape dynamic content from websites that are using AJAX?

by

Here is a simple example of scrapy with an AJAX request. Let see the site rubin-kazan.ru.

All messages are loaded with an AJAX request. My goal is to fetch these messages with all their attributes (author, date, …):

enter image description here

When I analyze the source code of the page I can’t see all these messages because the web page uses AJAX technology. But I can with Firebug from Mozilla Firefox (or an equivalent tool in other browsers) to analyze the HTTP request that generate the messages on the web page:

enter image description here

It doesn’t reload the whole page but only the parts of the page that contain messages. For this purpose I click an arbitrary number of page on the bottom:

enter image description here

And I observe the HTTP request that is responsible for message body:

enter image description here

After finish, I analyze the headers of the request (I must quote that this URL I’ll extract from source page from var section, see the code below):

enter image description here

And the form data content of the request (the HTTP method is “Post”):

enter image description here

And the content of response, which is a JSON file:

enter image description here

Which presents all the information I’m looking for.

From now, I must implement all this knowledge in scrapy. Let’s define the spider for this purpose:

class spider(BaseSpider):
    name="RubiGuesst"
    start_urls = ['http://www.rubin-kazan.ru/guestbook.html']

    def parse(self, response):
        url_list_gb_messages = re.search(r'url_list_gb_messages="(.*)"', response.body).group(1)
        yield FormRequest('http://www.rubin-kazan.ru' + url_list_gb_messages, callback=self.RubiGuessItem,
                          formdata={'page': str(page + 1), 'uid': ''})

    def RubiGuessItem(self, response):
        json_file = response.body

In parse function I have the response for first request.
In RubiGuessItem I have the JSON file with all information.

More Related Contents:

  1. scrape html generated by javascript with python
  2. Headless Browser for Python (Javascript support REQUIRED!) [closed]
  3. CORS error on same domain?
  4. Executing Javascript from Python
  5. Scraping ajax pages using python
  6. Javascript – No ‘Access-Control-Allow-Origin’ header is present on the requested resource
  7. Solve Cross Origin Resource Sharing with Flask
  8. Click on a javascript link within python?
  9. Get data from html and and pass the data back to the front end using ajax or js
  10. Python Flask Cors Issue
  11. Better to have ajax inside or outside for-loop or for-loop at the server side? Javascript [closed]
  12. How to manage a redirect request after a jQuery Ajax call
  13. JavaScript raises SyntaxError with data rendered in Jinja template
  14. Upload file with Ajax XMLHttpRequest
  15. URL Encode a string in jQuery for an AJAX request
  16. Download pdf file using jquery ajax
  17. How to refresh page on back button click?
  18. How to load json into my angular.js ng-model?
  19. how to assign javascript variable value to php variable
  20. How to send multipart/form-data form content by ajax (no jquery)?
  21. Dynamically refresh a part of the template when a variable is updated golang
  22. $http.get(…).success is not a function
  23. Why my $.ajax showing “preflight is invalid redirect error”?
  24. Looping over urls to do the same thing
  25. How can I get all a form’s values that would be submitted without submitting [duplicate]
  26. responseXML always null
  27. How to upload string as file with jQuery or other js framework
  28. how to schedule ajax calls every N seconds?
  29. XMLHttpRequest: Network Error 0x80070005, Access is denied on Microsoft Edge (but not IE)
  30. Phantom JS synchronous AJAX request : NETWORK_ERR: XMLHttpRequest Exception 101

Leave a Comment