How to fetch a non-ascii url with urlopen?

by

Strictly speaking URIs can’t contain non-ASCII characters; what you have there is an IRI.

To convert an IRI to a plain ASCII URI:

  • non-ASCII characters in the hostname part of the address have to be encoded using the Punycode-based IDNA algorithm;

  • non-ASCII characters in the path, and most of the other parts of the address have to be encoded using UTF-8 and %-encoding, as per Ignacio’s answer.

So:

import re, urlparse

def urlEncodeNonAscii(b):
    return re.sub('[\x80-\xFF]', lambda c: '%%%02x' % ord(c.group(0)), b)

def iriToUri(iri):
    parts= urlparse.urlparse(iri)
    return urlparse.urlunparse(
        part.encode('idna') if parti==1 else urlEncodeNonAscii(part.encode('utf-8'))
        for parti, part in enumerate(parts)
    )

>>> iriToUri(u'http://www.a\u0131b.com/a\u0131b')
'http://www.xn--ab-hpa.com/a%c4%b1b'

(Technically this still isn’t quite good enough in the general case because urlparse doesn’t split away any user:pass@ prefix or :port suffix on the hostname. Only the hostname part should be IDNA encoded. It’s easier to encode using normal urllib.quote and .encode('idna') at the time you’re constructing a URL than to have to pull an IRI apart.)

More Related Contents:

  1. urllib2 read to Unicode
  2. How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup? [duplicate]
  3. matching unicode characters in python regular expressions
  4. How can I speed up fetching pages with urllib2 in python?
  5. Tell urllib2 to use custom DNS
  6. UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xa0′ in position 20: ordinal not in range(128)
  7. Saving utf-8 texts with json.dumps as UTF8, not as \u escape sequence
  8. u’\ufeff’ in Python string
  9. Writing Unicode text to a text file?
  10. Proxy with urllib2
  11. How can I convert surrogate pairs to normal string in Python?
  12. Windows cmd encoding change causes Python crash
  13. Python Unicode Encode Error
  14. Using an HTTP PROXY – Python [duplicate]
  15. UnicodeDecodeError when redirecting to file
  16. Let JSON object accept bytes or let urlopen output strings
  17. Extracting text from script tag using BeautifulSoup in Python
  18. ElementTree and unicode
  19. Why does ENcoding a string result in a DEcoding error (UnicodeDecodeError)?
  20. “SyntaxError: Non-ASCII character …” or “SyntaxError: Non-UTF-8 code starting with …” trying to use non-ASCII text in a Python script
  21. Convert UTF-16 to UTF-8 and remove BOM?
  22. Difference between open and codecs.open in Python
  23. python encoding utf-8
  24. Download and decompress gzipped file in memory?
  25. How to save “complete webpage” not just basic html using Python
  26. Whitespace gone from PDF extraction, and strange word interpretation
  27. Unicode encoding for filesystem in Mac OS X not correct in Python?
  28. Chinese unicode fonts in PyGame
  29. Printing Unicode in eclipse Pydev console and in Idle
  30. UnicodeDecodeError when reading CSV file in Pandas

Leave a Comment