web scraping dynamic content with python

by

Instead of trying to reverse engineer it, you can use ghost.py to directly interact with JavaScript on the page.

If you run the following query in a chrome console, you’ll see it returns everything you want.

document.getElementsByClassName('inline-text-org');

Returns

[<div class=​"inline-text-org" title=​"University of Manchester">​University of Manchester​</div>, 
 <div class=​"inline-text-org" title=​"University of California Irvine">​University of California ...​</div>​
  etc...

You can run JavaScript through python in a real life DOM using ghost.py.

This is really cool:

from ghost import Ghost
ghost = Ghost()
page, resources = ghost.open('http://academic.research.microsoft.com/Search?query=lander')
result, resources = ghost.evaluate(
    "document.getElementsByClassName('inline-text-org');")

More Related Contents:

  1. Web scraping with Python [closed]
  2. Scraping ajax pages using python
  3. unable to call firefox from selenium in python on AWS machine
  4. how to scrape web page that is not written directly using HTML, but is auto-generated using JavaScript? [closed]
  5. Difference between text and innerHTML using Selenium
  6. selenium with scrapy for dynamic page
  7. How to use Python requests to fake a browser visit a.k.a and generate User Agent?
  8. can we use XPath with BeautifulSoup?
  9. Using python Requests with javascript pages
  10. Scraping dynamic content using python-Scrapy
  11. How to run Scrapy from within a Python script
  12. Web scraping program cannot find element which I can see in the browser
  13. ‘list’ object has no attribute ‘get_attribute’ while iterating through WebElements
  14. UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xe9’ – -when using urlib.request python3
  15. Beautiful Soup 4 find_all don’t find links that Beautiful Soup 3 finds
  16. How to click a link by text with No Text in Python
  17. Selenium Python: How to web scrape the element text
  18. How to get text from span tag in BeautifulSoup
  19. pandas read_html ValueError: No tables found
  20. How to scrape a public tableau dashboard? [closed]
  21. How to get the scrapy failure URLs?
  22. Fetch all href link using selenium in python
  23. Python regular expression for HTML parsing
  24. Exclude unwanted tag on Beautifulsoup Python
  25. Python – make a POST request using Python 3 urllib
  26. csv.writer writing each character of word in separate column/cell
  27. Speeding up beautifulsoup
  28. Scrape Dynamic contents created by Javascript using Python
  29. BeautifulSoup returns None even though the element exists
  30. Scraping Javascript driven web pages with PyQt4 – how to access pages that need authentication?

Leave a Comment