Parsing compressed xml feed into ElementTree

by

You can pass the value returned by urlopen() directly to GzipFile() and in turn you can pass it to ElementTree methods such as iterparse():

#!/usr/bin/env python3
import xml.etree.ElementTree as etree
from gzip import GzipFile
from urllib.request import urlopen, Request

with urlopen(Request("http://smarkets.s3.amazonaws.com/oddsfeed.xml",
                     headers={"Accept-Encoding": "gzip"})) as response, \
     GzipFile(fileobj=response) as xml_file:
    for elem in getelements(xml_file, 'interesting_tag'):
        process(elem)

where getelements() allows to parse files that do not fit in memory.

def getelements(filename_or_file, tag):
    """Yield *tag* elements from *filename_or_file* xml incrementaly."""
    context = iter(etree.iterparse(filename_or_file, events=('start', 'end')))
    _, root = next(context) # get root element
    for event, elem in context:
        if event == 'end' and elem.tag == tag:
            yield elem
            root.clear() # free memory

To preserve memory, the constructed xml tree is cleared on each tag element.

More Related Contents:

  1. Parsing XML with namespace in Python via ‘ElementTree’
  2. Using Python Iterparse For Large XML Files
  3. How do I get Python’s ElementTree to pretty print to an XML file?
  4. Can ElementTree be told to preserve the order of attributes?
  5. How to create with Python’s cElementTree
  6. Empty list returned from ElementTree findall
  7. Remove whitespaces in XML string
  8. parsing xml containing default namespace to get an element value using lxml
  9. Using XPath in ElementTree
  10. parsing XML file gets UnicodeEncodeError (ElementTree) / ValueError (lxml)
  11. How to write XML declaration using xml.etree.ElementTree
  12. Faithfully Preserve Comments in Parsed XML
  13. Converting xml to dictionary using ElementTree
  14. Is there a way to get a line number from an ElementTree Element
  15. Find element by text with XPath in ElementTree
  16. Merge xml files with nested elements without external libraries
  17. How do I parse and write XML using Python’s ElementTree without moving namespaces around?
  18. Use xml.etree.ElementTree to print nicely formatted xml files [duplicate]
  19. Emitting namespace specifications with ElementTree in Python
  20. ElementTree findall() returning empty list
  21. TypeError: ‘str’ does not support the buffer interface
  22. How to convert an XML file to nice pandas dataframe?
  23. How to use XPath in Python?
  24. access ElementTree node parent node
  25. What is the fastest way to parse large XML docs in Python?
  26. How to use Python XML findall to find ”
  27. How can I access namespaced XML elements using BeautifulSoup?
  28. Saving XML files using ElementTree
  29. How do I gzip compress a string in Python?
  30. Why is lxml.etree.iterparse() eating up all my memory?

Leave a Comment