What is the fastest way to parse large XML docs in Python?

by

I looks to me as if you do not need any DOM capabilities from your program. I would second the use of the (c)ElementTree library. If you use the iterparse function of the cElementTree module, you can work your way through the xml and deal with the events as they occur.

Note however, Fredriks advice on using cElementTree iterparse function:

to parse large files, you can get rid of elements as soon as you’ve processed them:

for event, elem in iterparse(source):
    if elem.tag == "record":
        ... process record elements ...
        elem.clear()

The above pattern has one drawback; it does not clear the root element, so you will end up with a single element with lots of empty child elements. If your files are huge, rather than just large, this might be a problem. To work around this, you need to get your hands on the root element. The easiest way to do this is to enable start events, and save a reference to the first element in a variable:

# get an iterable
context = iterparse(source, events=("start", "end"))

# turn it into an iterator
context = iter(context)

# get the root element
event, root = context.next()

for event, elem in context:
    if event == "end" and elem.tag == "record":
        ... process record elements ...
        root.clear()

The lxml.iterparse() does not allow this.

The previous does not work on Python 3.7, consider the following way to get the first element.

import xml.etree.ElementTree as ET

# Get an iterable.
context = ET.iterparse(source, events=("start", "end"))
    
for index, (event, elem) in enumerate(context):
    # Get the root element.
    if index == 0:
        root = elem
    if event == "end" and elem.tag == "record":
        # ... process record elements ...
        root.clear()

More Related Contents:

  1. How to convert an XML file to nice pandas dataframe?
  2. using lxml and iterparse() to parse a big (+- 1Gb) XML file
  3. Empty list returned from ElementTree findall
  4. Python code to remove HTML tags from a string [duplicate]
  5. Parsing a YAML file in Python, and accessing the data?
  6. Python BeautifulSoup XML Parsing
  7. Simple CSV to XML Conversion – Python
  8. Comparing two xml files in python
  9. Pretty print in lxml is failing when I add tags to a parsed tree
  10. Why is ElementTree raising a ParseError?
  11. Is doing multiplication multiple times or assigning to a variable faster?
  12. Parsing XML with namespace in Python via ‘ElementTree’
  13. Only extracting text from this element, not its children
  14. How do convert a pandas dataframe to XML?
  15. ‘module’ object has no attribute ‘loads’ while parsing JSON using python
  16. Speed comparison with Project Euler: C vs Python vs Erlang vs Haskell
  17. How do I measure elapsed time in Python?
  18. Remove namespace and prefix from xml in python using lxml
  19. Performance with global variables vs local
  20. How to read specific part of large file in Python
  21. Complexity of list.index(x) in Python
  22. pandas read_html ValueError: No tables found
  23. How does python and the regex module handle backslashes?
  24. What is the fastest way to output large DataFrame into a CSV file?
  25. How expensive are Python dictionaries to handle?
  26. Parsing a date that can be in several formats in python
  27. Parsing compressed xml feed into ElementTree
  28. differences between “d = dict()” and “d = {}”
  29. Split string at commas except when in bracket environment
  30. numpy faster than numba and cython , how to improve numba code

Leave a Comment