Unzipping part of a .gz file using python

by

The issue with the gzip module is not that it can’t decompress the partial file, the error occurs only at the end when it tries to verify the checksum of the decompressed content. (The original checksum is stored at the end of the compressed file so the verification will never, ever work with a partial file.)

The key is to trick gzip into skipping the verification. The answer by caesar0301 does this by modifying the gzip source code, but it’s not necessary to go that far, simple monkey patching will do. I wrote this context manager to temporarily replace gzip.GzipFile._read_eof while I decompress the partial file:

import contextlib

@contextlib.contextmanager
def patch_gzip_for_partial():
    """
    Context manager that replaces gzip.GzipFile._read_eof with a no-op.

    This is useful when decompressing partial files, something that won't
    work if GzipFile does it's checksum comparison.

    """
    _read_eof = gzip.GzipFile._read_eof
    gzip.GzipFile._read_eof = lambda *args, **kwargs: None
    yield
    gzip.GzipFile._read_eof = _read_eof

An example usage:

from cStringIO import StringIO

with patch_gzip_for_partial():
    decompressed = gzip.GzipFile(StringIO(compressed)).read()

More Related Contents:

  1. zlib.error: Error -3 while decompressing: incorrect header check
  2. Python decompressing gzip chunk-by-chunk
  3. TypeError: ‘str’ does not support the buffer interface
  4. Does python urllib2 automatically uncompress gzip data fetched from webpage?
  5. no module named zlib
  6. Python decompression relative performance?
  7. How to unzip gz file using Python
  8. Download and decompress gzipped file in memory?
  9. Installing Python-2.7 on Ubuntu 10.4
  10. Parsing compressed xml feed into ElementTree
  11. How do I gzip compress a string in Python?
  12. sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings
  13. Get uncompressed size of a .gz file in python
  14. python 2.7 script to add a column
  15. Separate a number into its digits and work with each digit
  16. Python 2: Conditional statement behaviour of lists with greater/smaller [duplicate]
  17. Lag when win.blit() background pygame
  18. Converting string with UTC offset to a datetime object [duplicate]
  19. Using Django time/date widgets in custom form
  20. Remove HTML tags not on an allowed list from a Python string
  21. How to make a 4d plot with matplotlib using arbitrary data
  22. Thread Safety in Python’s dictionary
  23. How can I use `return` to get back multiple values from a for loop? Can I put them in a list?
  24. StaleElementReferenceException on Python Selenium
  25. ‘list’ object has no attribute ‘get_attribute’ while iterating through WebElements
  26. How to export a table dataframe in PySpark to csv?
  27. OpenCV for Python 3.x under Windows [duplicate]
  28. Generate all possible lists of length N that sum to S in Python
  29. Python – Detect a QR code from an image and crop using OpenCV
  30. AND/OR in Python? [duplicate]

Leave a Comment