Scrape the absolute URL instead of a relative path in python

by

urllib.parse.urljoin() might help. It does a join, but it is smart about it and handles both relative and absolute paths. Note this is python 3 code.

>>> import urllib.parse
>>> base="https://www.example-page-xl.com"

>>> urllib.parse.urljoin(base, "https://stackoverflow.com/helloworld/index.php") 
'https://www.example-page-xl.com/helloworld/index.php'

>>> urllib.parse.urljoin(base, 'https://www.example-page-xl.com/helloworld/index.php')
'https://www.example-page-xl.com/helloworld/index.php'

More Related Contents:

  1. How to handle IncompleteRead: in python
  2. retrieve links from web page using python and BeautifulSoup [closed]
  3. UnicodeEncodeError: ‘charmap’ codec can’t encode characters
  4. Beautiful Soup: ‘ResultSet’ object has no attribute ‘find_all’?
  5. BeautifulSoup Grab Visible Webpage Text
  6. python BeautifulSoup parsing table
  7. How to avoid HTTP error 429 (Too Many Requests) python
  8. Only extracting text from this element, not its children
  9. Can I remove script tags with BeautifulSoup?
  10. Difference between “findAll” and “find_all” in BeautifulSoup
  11. Remove a tag using BeautifulSoup but keep its contents
  12. bs4.FeatureNotFound: Couldn’t find a tree builder with the features you requested: lxml. Do you need to install a parser library?
  13. BeautifulSoup webscraping find_all( ): finding exact match
  14. BeautifulSoup innerhtml?
  15. Using BeautifulSoup to find a HTML tag that contains certain text
  16. How can I insert a new tag into a BeautifulSoup object?
  17. Why do I get a recursion error with BeautifulSoup and IDLE?
  18. Beautiful Soup and extracting a div and its contents by ID
  19. What should I do if socket.setdefaulttimeout() is not working?
  20. How to extract a JSON object that was defined in a HTML page javascript block using Python?
  21. Matching partial ids in BeautifulSoup
  22. BeautifulSoup – modifying all links in a piece of HTML?
  23. Python regular expression for HTML parsing
  24. Python BeautifulSoup: wildcard attribute/id search
  25. How to install beautiful soup 4 with python 2.7 on windows
  26. how to get text from within a tag, but ignore other child tags
  27. PyQt Class not working for the second usage
  28. How to download a full webpage with a Python script?
  29. BeautifulSoup returns None even though the element exists
  30. Extract content within a tag with BeautifulSoup

Leave a Comment