Parsing broken XML with lxml.etree.iterparse

by

Edit:

This is an older answer and I would have done it differently today. And I’m not just referring to the dumb snark … since then BeutifulSoup4 is available and it’s really quite nice. I recommend that to anyone who stumbles over here.

The currently accepted answer is, well, not what one should do.
The question itself also has a bad assumption:

parser = lxml.etree.XMLParser(recover=True) #recovers from bad characters.

Actually recover=True is for recovering from misformed XML. There is however an “encoding” option which would have fixed your issue. 

parser = lxml.etree.XMLParser(encoding='utf-8' #Your encoding issue.
                              recover=True, #I assume you probably still want to recover from bad xml, it's quite nice. If not, remove.
                              )

That’s it, that’s the solution.

BTW — For anyone struggling with parsing XML in python, especially from third party sources. I know, I know, the documentation is bad and there are a lot of SO red herrings; a lot of bad advice.

  • lxml.etree.fromstring()? – That’s for perfectly formed XML, silly
  • BeautifulStoneSoup? – Slow, and has a way-stupid policy for self
    closing tags
  • lxml.etree.HTMLParser()? – (because the xml is broken)
    Here’s a secret – HTMLParser() is… a Parser with recover=True
  • lxml.html.soupparser? – The encoding detection is supposed to be better, but it has the same failings of BeautifulSoup for self closing tags. Perhaps you can combine XMLParser with BeautifulSoup’s UnicodeDammit
  • UnicodeDammit and other cockamamie stuff to fix encodings? – Well, UnicodeDammit is kind of cute, I like the name and it’s useful for stuff beyond xml, but things are usually fixed if you do the right thing with XMLParser()

You could be trying all sorts of stuff from what’s available online. lxml documentation could be better. The code above is what you need for 90% of your XML parsing cases. Here I’ll restate it:

magical_parser = XMLParser(encoding='utf-8', recover=True)
tree = etree.parse(StringIO(your_xml_string), magical_parser) #or pass in an open file object

You’re welcome. My headaches == your sanity. Plus it has other features you might need for, you know, XML.

More Related Contents:

  1. Using Python Iterparse For Large XML Files
  2. how to remove an element in lxml
  3. How can this function be rewritten to implement OrderedDict? [duplicate]
  4. Remove namespace and prefix from xml in python using lxml
  5. using lxml and iterparse() to parse a big (+- 1Gb) XML file
  6. How do I use a default namespace in an lxml xpath query?
  7. parsing xml containing default namespace to get an element value using lxml
  8. Parse SGML with Open Arbitrary Tags in Python 3
  9. parsing XML file gets UnicodeEncodeError (ElementTree) / ValueError (lxml)
  10. python – lxml: enforcing a specific order for attributes
  11. How to include the namespaces into a xml file using lxml?
  12. Why does this xpath fail using lxml in python?
  13. Pretty print in lxml is failing when I add tags to a parsed tree
  14. Why doesn’t xpath work when processing an XHTML document with lxml (in python)?
  15. Why is lxml.etree.iterparse() eating up all my memory?
  16. lxml: add namespace to input file
  17. How to convert an XML file to nice pandas dataframe?
  18. How to use XPath in Python?
  19. libxml install error using pip
  20. What is the fastest way to parse large XML docs in Python?
  21. Can ElementTree be told to preserve the order of attributes?
  22. How to use Python XML findall to find ”
  23. How can I convert XML into a Python object?
  24. How can I access namespaced XML elements using BeautifulSoup?
  25. Installing lxml module in python
  26. How to Pretty Print HTML to a file, with indentation
  27. Equivalent to InnerHTML when using lxml.html to parse HTML
  28. How to remove elements from XML using Python
  29. python: xml.etree.ElementTree, removing “namespaces”
  30. Why is ElementTree raising a ParseError?

Leave a Comment