Merging multiple files into one within Hadoop

by

In order to keep everything on the grid use hadoop streaming with a single reducer and cat as the mapper and reducer (basically a noop) – add compression using MR flags.

hadoop jar \
    $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar \<br>
    -Dmapred.reduce.tasks=1 \
    -Dmapred.job.queue.name=$QUEUE \
    -input "$INPUT" \
    -output "$OUTPUT" \
    -mapper cat \
    -reducer cat

If you want compression add
-Dmapred.output.compress=true \
-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec

More Related Contents:

  1. Pig Latin: Load multiple files from a date range (part of the directory structure)
  2. Difference between Pig and Hive? Why have both? [closed]
  3. Information about big data and hadoop [closed]
  4. Container is running beyond memory limits
  5. Difference between Hive internal tables and external tables?
  6. How to transpose/pivot data in hive?
  7. Name node is in safe mode. Not able to leave
  8. How to fix corrupt HDFS FIles
  9. Is it better to use the mapred or the mapreduce package to create a Hadoop Job?
  10. Chaining multiple MapReduce jobs in Hadoop
  11. How to delete and update a record in Hive
  12. Pyspark: get list of files/directories on HDFS path
  13. Hive unable to manually set number of reducers
  14. how many mappers and reduces will get created for a partitoned table in hive
  15. Namenode not getting started
  16. Hive: Add partitions for existing folder structure
  17. How to load data to hive from HDFS without removing the source file?
  18. Spark iterate HDFS directory
  19. Loading Data from a .txt file to Table Stored as ORC in Hive
  20. What is the use of grouping comparator in hadoop map reduce
  21. Default number of reducers
  22. Create HIVE Table with multi character delimiter
  23. Job queue for Hive action in oozie
  24. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  25. Datanode process not running in Hadoop
  26. Cannot Read a file from HDFS using Spark
  27. What is Google’s Dremel? How is it different from Mapreduce?
  28. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?
  29. Behavior of the parameter “mapred.min.split.size” in HDFS
  30. Hive Data Retrieval Queries: Difference between CLUSTER BY, ORDER BY, and SORT BY

Leave a Comment