Jsoup get dynamically generated HTML

by

JSoup has some basic connection handling included, but it is not a web browser. It excels at parsing static html content. It does not run any javascript, so you are out of luck. However, there are different options that you might follow:

  1. You can analyze the page that you want to retrieve and find out how the content you are interested in gets loaded. Often it is not very hard to tap the original source of the loaded content and work with this. This approach has the benefit that you get what you want with no need of extra libraries and the retrieval will be fast.

  2. You can use a (full) browser and automate the loading of the page. A very good tool for this is selenium webdriver in combination with the headless webkit browser phantomjs. This however requires extra software and extra libraries in your project and will run much much slower than the first solution.

More Related Contents:

  1. Page content is loaded with JavaScript and Jsoup doesn’t see it
  2. Javascript parser for Java [closed]
  3. Getting Jsoup to support dynamically generated html by JavaScript
  4. How to read XML using XPath in Java
  5. Parse a URI String into Name-Value Collection
  6. “Expected BEGIN_OBJECT but was STRING at line 1 column 1”
  7. How can I efficiently parse HTML with Java?
  8. How to parse JSON in Java
  9. How do I preserve line breaks when using jsoup to convert html to plain text?
  10. Date object SimpleDateFormat not parsing timestamp string correctly in Java (Android) environment
  11. Parsing JSON array into java.util.List with Gson
  12. Custom JSON deserializer using Gson
  13. How to decode JSON with unknown field using Gson?
  14. calling java methods in javascript code [closed]
  15. Parsing JSON string in Java
  16. Using Joda Date & Time API to parse multiple formats
  17. Execute managebean method from javascript onload event
  18. Java 8 LocalDateTime is parsing invalid date
  19. Parsing PDF files (especially with tables) with PDFBox
  20. Parse JavaScript with jsoup
  21. “Content is not allowed in prolog” when parsing perfectly valid XML on GAE
  22. Parsing a fixed-width formatted file in Java
  23. How to add proxy support to Jsoup?
  24. Detecting Device Type in a web application
  25. How to use HTML and CSS as a Java application GUI?
  26. Reading HTML file to DOM tree using Java
  27. Parse String date in (yyyy-MM-dd) format
  28. Simplest way to correctly load html from web page into a string in Java
  29. Android java.lang.NoClassDefFoundError: org.jsoup.Jsoup
  30. Java 8 LocalDateTime is parsing invalid datetime

Leave a Comment