parsing XML file gets UnicodeEncodeError (ElementTree) / ValueError (lxml)

by

You are using the decoded unicode value. Use r.raw raw response data instead:

r = requests.get(url, params=payload, stream=True)
r.raw.decode_content = True
etree.parse(r.raw)

which will read the data from the response directly; do note the stream=True option to .get().

Setting the r.raw.decode_content = True flag ensures that the raw socket will give you the decompressed content even if the response is gzip or deflate compressed.

You don’t have to stream the response; for smaller XML documents it is fine to use the response.content attribute, which is the un-decoded response body:

r = requests.get(url, params=payload)
xml = etree.fromstring(r.content)

XML parsers always expect bytes as input as the XML format itself dictates how the parser is to decode those bytes to Unicode text.

More Related Contents:

  1. Using Python Iterparse For Large XML Files
  2. parsing xml containing default namespace to get an element value using lxml
  3. Parsing XML with namespace in Python via ‘ElementTree’
  4. how to remove an element in lxml
  5. How can this function be rewritten to implement OrderedDict? [duplicate]
  6. Can ElementTree be told to preserve the order of attributes?
  7. Remove namespace and prefix from xml in python using lxml
  8. using lxml and iterparse() to parse a big (+- 1Gb) XML file
  9. How to create with Python’s cElementTree
  10. Python Requests package: Handling xml response
  11. Empty list returned from ElementTree findall
  12. How do I use a default namespace in an lxml xpath query?
  13. Using XPath in ElementTree
  14. Parse SGML with Open Arbitrary Tags in Python 3
  15. How to write XML declaration using xml.etree.ElementTree
  16. lxml etree xmlparser remove unwanted namespace
  17. python – lxml: enforcing a specific order for attributes
  18. Faithfully Preserve Comments in Parsed XML
  19. How to include the namespaces into a xml file using lxml?
  20. Converting xml to dictionary using ElementTree
  21. Is there a way to get a line number from an ElementTree Element
  22. Parsing compressed xml feed into ElementTree
  23. Why doesn’t xpath work when processing an XHTML document with lxml (in python)?
  24. Find element by text with XPath in ElementTree
  25. Merge xml files with nested elements without external libraries
  26. Why is lxml.etree.iterparse() eating up all my memory?
  27. Parsing broken XML with lxml.etree.iterparse
  28. Use xml.etree.ElementTree to print nicely formatted xml files [duplicate]
  29. Emitting namespace specifications with ElementTree in Python
  30. ElementTree findall() returning empty list

Leave a Comment