Download all pdf files from a website using Python

by

Check out the following implementation. I’ve used requests module instead of urllib to do the download. Moreover, I’ve used .select() method instead of .find_all() to avoid using re. 

import os
import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup

url = "http://www.gatsby.ucl.ac.uk/teaching/courses/ml1-2016.html"

#If there is no such folder, the script will create one automatically
folder_location = r'E:\webscraping'
if not os.path.exists(folder_location):os.mkdir(folder_location)

response = requests.get(url)
soup= BeautifulSoup(response.text, "html.parser")     
for link in soup.select("a[href$='.pdf']"):
    #Name the pdf files using the last portion of each link which are unique in this case
    filename = os.path.join(folder_location,link['href'].split("https://stackoverflow.com/")[-1])
    with open(filename, 'wb') as f:
        f.write(requests.get(urljoin(url,link['href'])).content)

More Related Contents:

  1. BeautifulSoup webscraping find_all( ): finding exact match
  2. Python regular expression for HTML parsing
  3. How to find elements by class
  4. can we use XPath with BeautifulSoup?
  5. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  6. Using BeautifulSoup to extract text without tags
  7. Scraping Google Finance (BeautifulSoup)
  8. How to scrape a website which requires login using python and beautifulsoup?
  9. BeautifulSoup – search by text inside a tag
  10. Web scraping program cannot find element which I can see in the browser
  11. What’s the cleanest way to extract URLs from a string using Python?
  12. UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xe9’ – -when using urlib.request python3
  13. Beautiful Soup 4 find_all don’t find links that Beautiful Soup 3 finds
  14. Using BeautifulSoup to find a HTML tag that contains certain text
  15. How to get text from span tag in BeautifulSoup
  16. BeautifulSoup returns empty list when searching by compound class names
  17. How to scrape only visible webpage text with BeautifulSoup?
  18. How to find tag with particular text with Beautiful Soup?
  19. Can bs4 get the dynamic content of a webpage if requests can’t?
  20. Extract the ‘src’ attribute from an ‘img’ tag using Beautiful Soup
  21. How find specific data attribute from html tag in BeautifulSoup4?
  22. Exclude unwanted tag on Beautifulsoup Python
  23. BeautifulSoup: Get the contents of a specific table
  24. Speeding up beautifulsoup
  25. Scrape Dynamic contents created by Javascript using Python
  26. BeautifulSoup returns None even though the element exists
  27. Clicking link using beautifulsoup in python
  28. Beautiful Soup and Table Scraping – lxml vs html parser
  29. How to scrape dynamic webpages by Python
  30. How to handle IncompleteRead: in python

Leave a Comment