How can I scrape a page with dynamic content (created by JavaScript) in Python?

by

EDIT Sept 2021: phantomjs isn’t maintained any more, either

EDIT 30/Dec/2017: This answer appears in top results of Google searches, so I decided to update it. The old answer is still at the end.

dryscape isn’t maintained anymore and the library dryscape developers recommend is Python 2 only. I have found using Selenium’s python library with Phantom JS as a web driver fast enough and easy to get the work done.

Once you have installed Phantom JS, make sure the phantomjs binary is available in the current path:

phantomjs --version
# result:
2.1.1

#Example
To give an example, I created a sample page with following HTML code. (link):

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Javascript scraping test</title>
</head>
<body>
  <p id='intro-text'>No javascript support</p>
  <script>
     document.getElementById('intro-text').innerHTML = 'Yay! Supports javascript';
  </script> 
</body>
</html>

without javascript it says: No javascript support and with javascript: Yay! Supports javascript

#Scraping without JS support:

import requests
from bs4 import BeautifulSoup
response = requests.get(my_url)
soup = BeautifulSoup(response.text)
soup.find(id="intro-text")
# Result:
<p id="intro-text">No javascript support</p>

#Scraping with JS support:

from selenium import webdriver
driver = webdriver.PhantomJS()
driver.get(my_url)
p_element = driver.find_element_by_id(id_='intro-text')
print(p_element.text)
# result:
'Yay! Supports javascript'

You can also use Python library dryscrape to scrape javascript driven websites.

#Scraping with JS support:

import dryscrape
from bs4 import BeautifulSoup
session = dryscrape.Session()
session.visit(my_url)
response = session.body()
soup = BeautifulSoup(response)
soup.find(id="intro-text")
# Result:
<p id="intro-text">Yay! Supports javascript</p>

More Related Contents:

  1. Web-scraping JavaScript page with Python
  2. How to convert raw javascript object to a dictionary?
  3. How do I call a Javascript function from Python?
  4. Can I scrape the raw data from highcharts.js?
  5. How do I find what is common between 2 lists? [duplicate]
  6. Error tells me I need JavaScript
  7. Javascript equivalent of Python’s zip function
  8. Django Template Variables and Javascript
  9. Running javascript in Selenium using Python
  10. CasperJS/PhantomJS doesn’t load https page
  11. Call Python function from JavaScript code
  12. Pass parameter with Python Flask in external Javascript
  13. Rendering to JS with Jinja produces invalid number rather than string
  14. Pass JavaScript variable to Flask url_for
  15. Python library for rendering HTML and javascript [closed]
  16. How can I handle Javascript in a Perl web crawler?
  17. How do I wait for a JavaScript __doPostBack call through Selenium and WebDriver
  18. How to call python script from NodeJs
  19. How to decrypt password from JavaScript CryptoJS.AES.encrypt(password, passphrase) in Python
  20. How can I pass parameter through serialization and then parse them using external JavaScript?
  21. How to get html with javascript rendered sourcecode by using selenium
  22. How do I download a file from FastAPI backend using Fetch API in the frontend?
  23. How do i take picture from client side(html) and save it to server side(Python)
  24. Splat operators in JavaScript, equivalent to *args and **kwargs in Python?
  25. Flask Dynamic data update without reload page
  26. What’s different between Python and Javascript regular expressions?
  27. FastAPI is not returning cookies to React frontend
  28. Passing variables from Flask to JavaScript
  29. How to send a base64 encoded image to a FastAPI backend?
  30. Get data from html and and pass the data back to the front end using ajax or js

Leave a Comment