Screen Scraping from a web page with a lot of Javascript [closed]

by

You may consider using HTMLunit
It’s a java class library made to automate browsing without having to control a browser, and it integrates the Mozilla Rhino Javascript engine to process javascript on the pages it loads. There’s also a JRuby wrapper for that, named Celerity. Its javascript support is not really perfect right now, but if your pages don’t use many hacks things should work fine the performance should be way better than controlling a browser. Furthermore, you don’t have to worry about cookies being persisted after your scraping is over and all the other nasty things connected to controlling a browser (history, autocomplete, temp files etc).

More Related Contents:

  1. Retrieve the position (X,Y) of an HTML element
  2. How can I tell if a DOM element is visible in the current viewport?
  3. Parse an HTML string with JS
  4. jQuery document.createElement equivalent?
  5. Browser detection versus feature detection
  6. How can I count text lines inside an DOM element? Can I?
  7. How do I create a link using javascript?
  8. How to show Page Loading div until the page has finished loading?
  9. Find the closest ancestor element that has a specific class
  10. Retrieving the text of the selected in element
  11. Replace HTML page with contents retrieved via AJAX
  12. How do I replace text inside a div element?
  13. Dynamically creating HTML elements using Javascript?
  14. What’s standard behavior when element is clicked? Will it submit the form?
  15. Get element at specified position – JavaScript
  16. Pass mouse events through absolutely-positioned element
  17. Add SVG element to existing SVG using DOM
  18. How to get the HTML for a DOM element in javascript
  19. How to find a parent with a known class in jQuery?
  20. How to iterate through all attributes in an HTML element?
  21. Why aren’t some technically serializable input properties serializable?
  22. HTML Table with vertical rows
  23. How to move table row in jQuery?
  24. Add an element to the DOM with JavaScript
  25. Is it possible to use HTML’s .querySelector() to select by xlink attribute in an SVG?
  26. How to get the title of HTML page with JavaScript?
  27. How to append text to a div element?
  28. JQuery html() vs. innerHTML
  29. Images failing to load in IE with DOM: 7009 error (unable to decode) in console
  30. Why is iframe.contentWindow == null?

Leave a Comment