Using python Requests with javascript pages

by

Good news: there is now a requests module that supports javascript: https://pypi.org/project/requests-html/

from requests_html import HTMLSession

session = HTMLSession()

r = session.get('http://www.yourjspage.com')

r.html.render()  # this call executes the js in the page

As a bonus this wraps BeautifulSoup, I think, so you can do things like

r.html.find('#myElementID').text

which returns the content of the HTML element as you’d expect.

More Related Contents:

  1. How to use Python requests to fake a browser visit a.k.a and generate User Agent?
  2. Web scraping program cannot find element which I can see in the browser
  3. Can bs4 get the dynamic content of a webpage if requests can’t?
  4. Python 3: using requests does not get the full content of a web page
  5. XHR request URL says does not exist when attempting to parse it’s content
  6. Correct way to try/except using Python requests module?
  7. Max retries exceeded with URL in requests
  8. Log all requests from the python-requests module
  9. Python Requests requests.exceptions.SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol
  10. Scraping Google Finance (BeautifulSoup)
  11. SSL InsecurePlatform error when using Requests package
  12. How to scrape a website which requires login using python and beautifulsoup?
  13. Python Requests – How to use system ca-certificates (debian/ubuntu)?
  14. How do I read a response from Python Requests?
  15. Python – requests.exceptions.SSLError – dh key too small
  16. Is the Session object from Python’s Requests library thread safe?
  17. Login to Facebook using python requests
  18. zsh: no matches found: requests[security]
  19. Requests failing to connect to a TLS server
  20. How to scrape only visible webpage text with BeautifulSoup?
  21. Scraping dynamic data selenium – Unable to locate element
  22. How to find tag with particular text with Beautiful Soup?
  23. Python requests library how to pass Authorization header with single token
  24. How find specific data attribute from html tag in BeautifulSoup4?
  25. How to save requests (python) cookies to a file?
  26. Saving response from Requests to file
  27. Python Requests – SSL error for client side cert
  28. Getting HEAD content with Python Requests
  29. Return a requests.Response object from Flask
  30. How to measure download speed and progress using requests?

Leave a Comment