Python 3: using requests does not get the full content of a web page

by

The page is rendered with JavaScript making more requests to fetch additional data. You can fetch the complete page with selenium.

from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome()
url = "https://shop.nordstrom.com/c/womens-dresses-shop?origin=topnav&cm_sp=Top%20Navigation-_-Women-_-Dresses&offset=11&page=3&top=72"
driver.get(url)
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()
print(soup.prettify())

For other solutions see my answer to Scraping Google Finance (BeautifulSoup)

More Related Contents:

  1. How to use Python requests to fake a browser visit a.k.a and generate User Agent?
  2. Using python Requests with javascript pages
  3. Web scraping program cannot find element which I can see in the browser
  4. Can bs4 get the dynamic content of a webpage if requests can’t?
  5. XHR request URL says does not exist when attempting to parse it’s content
  6. How to POST JSON data with Python Requests?
  7. How to install packages offline?
  8. How to send a “multipart/form-data” with requests in python?
  9. Sending “User-agent” using Requests library in Python
  10. Scraping dynamic content using python-Scrapy
  11. How can I download a file on a click event using selenium?
  12. How do I disable log messages from the Requests library?
  13. How to save an image locally using Python whose URL address I already know?
  14. Python requests.exceptions.SSLError: EOF occurred in violation of protocol
  15. “SSL: certificate_verify_failed” error when scraping https://www.thenewboston.com/
  16. Download and save PDF file with Python requests module
  17. Python ‘requests’ library – define specific DNS?
  18. requests: how to disable / bypass proxy
  19. How can I send an xml body using requests library?
  20. Putting a `Cookie` in a `CookieJar`
  21. Requests library: missing SSL handshake certificates file after cx_Freeze
  22. google search with python requests library
  23. How to get the scrapy failure URLs?
  24. SSL error with Python requests despite up-to-date dependencies
  25. python requests.get always get 404
  26. What is the proper way to make downstream Https requests inside of Uvicorn/FastAPI?
  27. Speeding up beautifulsoup
  28. How to limit download rate of HTTP requests in requests python library?
  29. Clicking link using beautifulsoup in python
  30. Beautiful Soup and Table Scraping – lxml vs html parser

Leave a Comment