How can I use Perl to grab text from a web page that is dynamically generated with JavaScript?

by

You’ll need to reverse-engineer what the Javascript is doing. Does it fire off an AJAX request to populate the <div>? If so, it should be pretty easy to sniff the request using Firebug and then duplicate it with LWP::UserAgent or WWW::Mechanize to get the information.

If the Javascript is just doing pure DOM manipulation, then that means the data must exist somewhere else in the page or the Javascript already. So figure out where it’s coming from and grab it.

Finally, if none of those options are adequate, you may need to just use a real browser to do it. There are a few options for automating browser behavior, like WWW::Mechanize::Firefox or Win32::IE::Mechanize.

More Related Contents:

  1. Submit form and stay on same page?
  2. Screen Scraping from a web page with a lot of Javascript [closed]
  3. Displaying selected dropdown option [closed]
  4. How could I make a calendar? [closed]
  5. Is it possible to apply CSS to half of a character?
  6. How can I know which radio button is selected via jQuery?
  7. Highlight a word with jQuery
  8. What is the most efficient way to create HTML elements using jQuery?
  9. How to check if a specific pixel of an image is transparent?
  10. How to use Checkbox inside Select Option
  11. Render HTML to an image
  12. Why is it bad practice to use links with the javascript: “protocol”?
  13. How to avoid the need for ctrl-click in a multi-select box using Javascript?
  14. Very Simple, Very Smooth, JavaScript Marquee
  15. Get JSON data from external URL and display it in a div as plain text
  16. Change color of specific words in textarea
  17. How to save image from canvas with CSS filters
  18. Wrap link around
  19. contentDocument for an iframe
  20. make checkbox behave like radio buttons with javascript
  21. Change iframe src by clicking a link
  22. How to generate a thumbnail image after adding an image inside an input type=”file” in a form and submitting them both on the same form
  23. Proper way to reset a GIF animation with display:none on Chrome
  24. Disable button whenever a text field is empty dynamically
  25. How do I select the innermost element?
  26. Trigger CSS Animations in JavaScript
  27. Dynamically change the size of the font size based on text length
  28. How do I capture a key press (or keydown) event on a div element?
  29. Large file upload with WebSocket
  30. Popstate on page’s load in Chrome

Leave a Comment