Wait page to load before getting data with requests.get in python 3

by

It doesn’t look like a problem of waiting, it looks like the element is being created by JavaScript, requests can’t handle dynamically generated elements by JavaScript. A suggestion is to use selenium together with PhantomJS to get the page source, then you can use BeautifulSoup for your parsing, the code shown below will do exactly that:

from bs4 import BeautifulSoup
from selenium import webdriver

url = "http://legendas.tv/busca/walking%20dead%20s03e02"
browser = webdriver.PhantomJS()
browser.get(url)
html = browser.page_source
soup = BeautifulSoup(html, 'lxml')
a = soup.find('section', 'wrapper')

Also, there’s no need to use .findAll if you are only looking for one element only.

More Related Contents:

  1. Scraping Google Finance (BeautifulSoup)
  2. Web scraping program cannot find element which I can see in the browser
  3. Failed to load resource: the server responded with a status of 429 (Too Many Requests) and 404 (Not Found) with ChromeDriver Chrome through Selenium
  4. “Eager” Page Load Strategy workaround for Chromedriver Selenium in Python
  5. Python SSL certificate verify error
  6. Can bs4 get the dynamic content of a webpage if requests can’t?
  7. How i can get new ip from tor every requests in threads?
  8. Scrape Dynamic contents created by Javascript using Python
  9. Fetching multiple urls with aiohttp in python
  10. Need help installing requests for python 3
  11. Writing a specific function in python
  12. Invalid selector: Compound class names not permitted error using Selenium
  13. webdriver.FirefoxProfile(): Is it possible to use a profile without making a copy of it?
  14. Why does loading tensorflow on Mac lead to “Process finished with exit code 132 (interrupted by signal 4: SIGILL)”?
  15. Print doesn’t print when it’s in map, Python
  16. seaborn histplot and displot output doesn’t match
  17. ImportError: No module named ‘psycopg2._psycopg’
  18. Error when executing `jupyter notebook` (No such file or directory)
  19. TypeError: can’t pickle _thread.lock objects
  20. Optimize Sieve of Eratosthenes Further
  21. How to replace the icon in a Tkinter app?
  22. How to click on the anchor element when the spinner obscures it using Selenium and Python?
  23. Is this possible to load the page after the javascript execute using python?
  24. How to make python 3 print() utf8
  25. Pandas pd.cut() – binning datetime column / series
  26. Permission Error When Trying to Use PyInstaller
  27. Keras, Tensorflow : Merge two different model output into one
  28. Clicking link using beautifulsoup in python
  29. Beautiful Soup and Table Scraping – lxml vs html parser
  30. Download all pdf files from a website using Python

Leave a Comment