Fetch contents(loaded through AJAX call) of a web page

by

Jsoup is a html parser only. Unfortunately it’s not possible to parse any javascript / ajax content, since jsoup can’t execute those.

The solution: using a library which can handle Scripts.

Here are some examples i know:

If such a library doesn’t support parsing or selectors, you can at least use them to get Html out of the scripts (which then can be parsed by jsoup).

More Related Contents:

  1. JSoup UserAgent, how to set it right?
  2. Using JSoup To Extract HTML Table Contents
  3. can someone explain what this code is doing
  4. Writing a web crawler in Java [closed]
  5. Sending “User-agent” using Requests library in Python
  6. Jsoup Java HTML parser : Executing Javascript events
  7. How to “scan” a website (or page) for info, and bring it into my program?
  8. How to run Scrapy from within a Python script
  9. How to connect via HTTPS using Jsoup?
  10. Jsoup get dynamically generated HTML
  11. Spider a Website and Return URLs Only
  12. How to use Jsoup with Volley?
  13. how to filter duplicate requests based on url in scrapy
  14. How to fill a form with Jsoup?
  15. Nodejs: Async request with a list of URL
  16. How to parse XML with jsoup
  17. Manipulating data on webs in Android
  18. How can I use different pipelines for different spiders in a single Scrapy project
  19. Python: maximum recursion depth exceeded while calling a Python object
  20. how do web crawlers handle javascript
  21. Web crawler that can interpret JavaScript [closed]
  22. Can Jsoup simulate a button press?
  23. Submit form with no submit button in rvest
  24. selenium implicitly wait doesn’t work
  25. How to write a crawler?
  26. Python: Disable images in Selenium Google ChromeDriver
  27. Locally run all of the spiders in Scrapy
  28. How to print html source to console with phantomjs
  29. Crawling the Google Play store
  30. Scrapy CrawlSpider doesn’t crawl the first landing page

Leave a Comment