How to scroll down with Phantomjs to load dynamic content

by

Found a way to do it and tried to adapt to your situation. I didn’t test the best way of finding the bottom of the page because I had a different context, but check the solution below. The thing here is that you have to wait a little for the page to load and javascript works asynchronously so you have to use setInterval or setTimeout (see) to achieve this.

page.open('http://example.com/?q=houston', function () {

  // Check for the bottom div and scroll down from time to time
  window.setInterval(function() {
      // Check if there is a div with class=".has-more-items" 
      // (not sure if there's a better way of doing this)
      var count = page.content.match(/class=".has-more-items"/g);

      if(count === null) { // Didn't find
        page.evaluate(function() {
          // Scroll to the bottom of page
          window.document.body.scrollTop = document.body.scrollHeight;
        });
      }
      else { // Found
        // Do what you want
        ...
        phantom.exit();
      }
  }, 500); // Number of milliseconds to wait between scrolls

});

More Related Contents:

  1. CasperJS/PhantomJS doesn’t load https page
  2. Save and render a webpage with PhantomJS and node.js
  3. Screen Scraping from a web page with a lot of Javascript [closed]
  4. Phantomjs page.content isn’t retrieving the page content
  5. Scrape web pages in real time with Node.js
  6. Looping over urls to do the same thing
  7. Parse an HTML string with JS
  8. jQuery document.createElement equivalent?
  9. When should I use jQuery’s document.ready function?
  10. How can I count text lines inside an DOM element? Can I?
  11. querySelector, wildcard element match?
  12. PhantomJS; click an element
  13. Copy / Put text on the clipboard with FireFox, Safari and Chrome
  14. How to serialize DOM node to JSON even if there are circular references?
  15. How do I replace text inside a div element?
  16. Dynamically creating HTML elements using Javascript?
  17. What’s standard behavior when element is clicked? Will it submit the form?
  18. How to bind bootstrap popover on dynamic elements
  19. getElementsByName in IE7
  20. Add SVG element to existing SVG using DOM
  21. How to access DOM elements in electron?
  22. Why aren’t some technically serializable input properties serializable?
  23. Forcing a DOM refresh in Internet explorer after javascript dom manipulation
  24. How to give a Blob uploaded as FormData a file name?
  25. Add an element to the DOM with JavaScript
  26. What is the difference between Document and document in JavaScript?
  27. Storing a variable in the JavaScript ‘window’ object is a proper way to use that object?
  28. To exceed the ImportXML limit on Google Spreadsheet
  29. How to prevent an iframe from reloading when moving it in the DOM
  30. Create a table in SVG

Leave a Comment