retrieve links from web page using python and BeautifulSoup [closed]

by

Here’s a short snippet using the SoupStrainer class in BeautifulSoup:

import httplib2
from bs4 import BeautifulSoup, SoupStrainer

http = httplib2.Http()
status, response = http.request('http://www.nytimes.com')

for link in BeautifulSoup(response, parse_only=SoupStrainer('a')):
    if link.has_attr('href'):
        print(link['href'])

The BeautifulSoup documentation is actually quite good, and covers a number of typical scenarios:

https://www.crummy.com/software/BeautifulSoup/bs4/doc/

Edit: Note that I used the SoupStrainer class because it’s a bit more efficient (memory and speed wise), if you know what you’re parsing in advance.

More Related Contents:

  1. How to find elements by class
  2. can we use XPath with BeautifulSoup?
  3. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  4. Using BeautifulSoup to extract text without tags
  5. Scraping Google Finance (BeautifulSoup)
  6. How to scrape a website which requires login using python and beautifulsoup?
  7. Web scraping program cannot find element which I can see in the browser
  8. How can I get href links from HTML using Python?
  9. BeautifulSoup webscraping find_all( ): finding exact match
  10. UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xe9’ – -when using urlib.request python3
  11. Beautiful Soup 4 find_all don’t find links that Beautiful Soup 3 finds
  12. How to get text from span tag in BeautifulSoup
  13. How to scrape only visible webpage text with BeautifulSoup?
  14. How to find tag with particular text with Beautiful Soup?
  15. Can bs4 get the dynamic content of a webpage if requests can’t?
  16. Python regular expression for HTML parsing
  17. How find specific data attribute from html tag in BeautifulSoup4?
  18. Exclude unwanted tag on Beautifulsoup Python
  19. BeautifulSoup: Get the contents of a specific table
  20. Speeding up beautifulsoup
  21. Scrape Dynamic contents created by Javascript using Python
  22. BeautifulSoup returns None even though the element exists
  23. Clicking link using beautifulsoup in python
  24. Beautiful Soup and Table Scraping – lxml vs html parser
  25. Download all pdf files from a website using Python
  26. How to scrape dynamic webpages by Python
  27. How to handle IncompleteRead: in python
  28. Reading dynamically generated web pages using python
  29. pandas read_html ValueError: No tables found
  30. Python BeautifulSoup: wildcard attribute/id search

Leave a Comment