Fast Concatenation of Multiple GZip Files

by

With gzip files, you can simply concatenate the files together, like so:

cat file1.gz file2.gz file3.gz > allfiles.gz

Per the gzip RFC,

A gzip file consists of a series of “members” (compressed data sets). […] The members simply appear one after another in the file, with no additional information before, between, or after them.

Note that this is not exactly the same as building a single gzip file of the concatenated data; among other things, all of the original filenames are preserved. However, gunzip seems to handle it as equivalent to a concatenation.

Since existing tools generally ignore the filename headers for the additional members, it’s not easily possible to extract individual files from the result. If you want this to be possible, build a ZIP file instead. ZIP and GZIP both use the DEFLATE algorithm for the actual compression (ZIP supports some other compression algorithms as well as an option – method 8 is the one that corresponds to GZIP’s compression); the difference is in the metadata format. Since the metadata is uncompressed, it’s simple enough to strip off the gzip headers and tack on ZIP file headers and a central directory record instead. Refer to the gzip format specification and the ZIP format specification.

More Related Contents:

  1. What do you think about the new Debian release cycle?
  2. Where can I set environment variables that crontab will use?
  3. How do I write standard error to a file while using “tee” with a pipe?
  4. Is it possible to use “/” in a filename?
  5. What is the meaning of “POSIX”?
  6. What does $@ mean in a shell script?
  7. How to convert DOS/Windows newline (CRLF) to Unix newline (LF)
  8. How do you find the original user through multiple sudo and su commands?
  9. How can I remove the last character of a file in unix?
  10. Managing log files created by cron jobs
  11. How to run a shell script on a Unix console or Mac terminal?
  12. What is a .pid file and what does it contain?
  13. Finding contents of one file in another file
  14. Difference between Real User ID, Effective User ID and Saved User ID
  15. Should PATH contain directories or full paths to binaries?
  16. Execute a command without keeping it in history [closed]
  17. How do I set up cron to run a file just once at a specific time? [closed]
  18. How can I pipe initial input into process which will then be interactive?
  19. How can I copy the output of a command directly into my clipboard?
  20. How do I get the path of a process in Unix / Linux?
  21. How to convert \uXXXX unicode to UTF-8 using console tools in *nix
  22. Running shell script in parallel
  23. Difference between shell and environment variables
  24. Why crontab uses OR when both day of month and day of week specified?
  25. What’s the point of eval/bash -c as opposed to just evaluating a variable?
  26. Forking vs Threading
  27. How to attach a file using mail command on Linux? [duplicate]
  28. What are the rules for valid identifiers (e.g. functions, vars, etc) in Bash?
  29. What is PATH on a Mac (UNIX) system?
  30. understanding and decoding the file mode value from stat function output

Leave a Comment