Regular expression to extract URL from an HTML link

by

If you’re only looking for one:

import re
match = re.search(r'href=[\'"]?([^\'" >]+)', s)
if match:
    print(match.group(1))

If you have a long string, and want every instance of the pattern in it:

import re
urls = re.findall(r'href=[\'"]?([^\'" >]+)', s)
print(', '.join(urls))

Where s is the string that you’re looking for matches in.

Quick explanation of the regexp bits:

r'...' is a “raw” string. It stops you having to worry about escaping characters quite as much as you normally would. (\ especially — in a raw string a \ is just a \. In a regular string you’d have to do \\ every time, and that gets old in regexps.)

“href=[\'"]?” says to match “href=”, possibly followed by a ' or ". “Possibly” because it’s hard to say how horrible the HTML you’re looking at is, and the quotes aren’t strictly required.

Enclosing the next bit in “()” says to make it a “group”, which means to split it out and return it separately to us. It’s just a way to say “this is the part of the pattern I’m interested in.”

“[^\'" >]+” says to match any characters that aren’t ', ", >, or a space. Essentially this is a list of characters that are an end to the URL. It lets us avoid trying to write a regexp that reliably matches a full URL, which can be a bit complicated.

The suggestion in another answer to use BeautifulSoup isn’t bad, but it does introduce a higher level of external requirements. Plus it doesn’t help you in your stated goal of learning regexps, which I’d assume this specific html-parsing project is just a part of.

It’s pretty easy to do:

from BeautifulSoup import BeautifulSoup
soup = BeautifulSoup(html_to_parse)
for tag in soup.findAll('a', href=True):
    print(tag['href'])

Once you’ve installed BeautifulSoup, anyway.

More Related Contents:

  1. Searching for IP addresses in a file
  2. How to understand regular expression with python?
  3. Fast punctuation removal with pandas
  4. Python regex matching Unicode properties
  5. Python: How to match nested parentheses with regex?
  6. How do you validate a URL with a regular expression in Python?
  7. How can I remove text within parentheses with a regex?
  8. python re.sub group: number after \number
  9. Handling backreferences to capturing groups in re.sub replacement pattern
  10. Python regular expressions – how to capture multiple groups from a wildcard expression?
  11. How to split by commas that are not within parentheses?
  12. Replace first occurrence of string in Python
  13. BeautifulSoup webscraping find_all( ): finding exact match
  14. How to handle response encoding from urllib.request.urlopen() , to avoid TypeError: can’t use a string pattern on a bytes-like object
  15. Regex: AttributeError: ‘NoneType’ object has no attribute ‘groups’
  16. remove unicode emoji using re in python
  17. Python – RegEx for splitting text into sentences (sentence-tokenizing) [duplicate]
  18. Re.match() returns always none
  19. How to use regular expression in lxml xpath?
  20. Why isn’t the regular expression’s “non-capturing” group working?
  21. Python Regular Expression Match All 5 Digit Numbers but None Larger
  22. Python regular expression for HTML parsing
  23. Remove Sub String by using Python
  24. How does python and the regex module handle backslashes?
  25. How to tell if one regular expression matches a subset of another regular expression?
  26. How do I verify that a string only contains letters, numbers, underscores and dashes? [duplicate]
  27. Python regex findall
  28. Regexp finding longest common prefix of two strings
  29. Splitting a math expression string into tokens in Python
  30. Download all pdf files from a website using Python

Leave a Comment