Hadoop: compress file in HDFS?

by

For me, it’s lower overhead to write a Hadoop Streaming job to compress files.

This is the command I run:

hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-0.20.2-cdh3u2.jar \
  -Dmapred.output.compress=true \
  -Dmapred.compress.map.output=true \
  -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec \
  -Dmapred.reduce.tasks=0 \
  -input <input-path> \
  -output $OUTPUT \
  -mapper "cut -f 2"

I’ll also typically stash the output in a temp folder in case something goes wrong:

OUTPUT=/tmp/hdfs-gzip-`basename $1`-$RANDOM

One additional note, I do not specify a reducer in the streaming job, but you certainly can. It will force all the lines to be sorted which can take a long time with a large file. There might be a way to get around this by overriding the partitioner but I didn’t bother figuring that out. The unfortunate part of this is that you potentially end up with many small files that do not utilize HDFS blocks efficiently. That’s one reason to look into Hadoop Archives

More Related Contents:

  1. How are zlib, gzip and zip related? What do they have in common and how are they different?
  2. What is a good Java library to zip/unzip files? [closed]
  3. Read whole text files from a compression in Spark
  4. Gzip versus minify
  5. Compression formats with good support for random access within archives? [closed]
  6. Node.js: Gzip compression?
  7. How to add .arc decompression to Inno Setup?
  8. How to enable GZIP compression in IIS 7.5
  9. How does Google’s Page Speed lossless image compression work?
  10. Best JavaScript compressor [closed]
  11. Difference between Hive internal tables and external tables?
  12. Name node is in safe mode. Not able to leave
  13. How to fix corrupt HDFS FIles
  14. Steganography in lossy compression (JAVA)
  15. How to delete and update a record in Hive
  16. How to implement GZip compression in ASP.NET?
  17. Reading file as single record in hadoop
  18. Hive unable to manually set number of reducers
  19. Compress numpy arrays efficiently
  20. MultipleOutputFormat in hadoop
  21. How to Access Hive via Python?
  22. Java compressing Strings
  23. Spark iterate HDFS directory
  24. Hadoop: …be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation
  25. Pig Latin: Load multiple files from a date range (part of the directory structure)
  26. How to list all files in a directory and its subdirectories in hadoop hdfs
  27. Spark: long delay between jobs
  28. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  29. Using SharpZipLib to unzip specific files?
  30. Are there any downsides to using UPX to compress a Windows executable?

Leave a Comment