urllib2 read to Unicode

by

After the operations you performed, you’ll see:

>>> req.headers['content-type']
'text/html; charset=windows-1251'

and so:

>>> encoding=req.headers['content-type'].split('charset=")[-1]
>>> ucontent = unicode(content, encoding)

ucontent is now a Unicode string (of 140655 characters) — so for example to display a part of it, if your terminal is UTF-8:

>>> print ucontent[76:110].encode("utf-8')
<title>Lenta.ru: Главное: </title>

and you can search, etc, etc.

Edit: Unicode I/O is usually tricky (this may be what’s holding up the original asker) but I’m going to bypass the difficult problem of inputting Unicode strings to an interactive Python interpreter (completely unrelated to the original question) to show how, once a Unicode string IS correctly input (I’m doing it by codepoints — goofy but not tricky;-), search is absolutely a no-brainer (and thus hopefully the original question has been thoroughly answered). Again assuming a UTF-8 terminal:

>>> x=u'\u0413\u043b\u0430\u0432\u043d\u043e\u0435'
>>> print x.encode('utf-8')
Главное
>>> x in ucontent
True
>>> ucontent.find(x)
93

Note: Keep in mind that this method may not work for all sites, since some sites only specify character encoding inside the served documents (using http-equiv meta tags, for example).

More Related Contents:

  1. How to fetch a non-ascii url with urlopen?
  2. How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup? [duplicate]
  3. What exactly do “u” and “r” string flags do, and what are raw string literals?
  4. Import error: No module name urllib2
  5. How to remove \xa0 from string in Python?
  6. How to get string objects instead of Unicode from JSON?
  7. Read and Write CSV files including unicode with Python 2.7
  8. Remove punctuation from Unicode formatted strings
  9. Why does Python print unicode characters when the default encoding is ASCII?
  10. Get a list of all the encodings Python can encode to
  11. What is the difference between encode/decode?
  12. python ignore certificate validation urllib2
  13. Python 3 print() function with Farsi/Arabic characters [duplicate]
  14. Python – Unicode to ASCII conversion
  15. UnicodeDecodeError: ‘utf8’ codec can’t decode bytes in position 3-6: invalid data
  16. What does ‘u’ mean in a list?
  17. Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence
  18. How is unicode represented internally in Python?
  19. Unicode identifiers in Python?
  20. open url from pythonanywhere
  21. Submitting to a web form using python
  22. Printing a string prints ‘u’ before the string in Python?
  23. Limiting number of processes in multiprocessing python
  24. Matching only a unicode letter in Python re
  25. How does unicodedata.normalize(form, unistr) work?
  26. Python – dealing with mixed-encoding files
  27. Python the same char not equals
  28. string.translate() with unicode data in python
  29. Python3 error: initial_value must be str or None, with StringIO
  30. Bytes in a unicode Python string

Leave a Comment