How to convert .txt file to Hadoop’s sequence file format

by

So the way more simplest answer is just an “identity” job that has a SequenceFile output.

Looks like this in java:

    public static void main(String[] args) throws IOException,
        InterruptedException, ClassNotFoundException {

    Configuration conf = new Configuration();
    Job job = new Job(conf);
    job.setJobName("Convert Text");
    job.setJarByClass(Mapper.class);

    job.setMapperClass(Mapper.class);
    job.setReducerClass(Reducer.class);

    // increase if you need sorting or a special number of files
    job.setNumReduceTasks(0);

    job.setOutputKeyClass(LongWritable.class);
    job.setOutputValueClass(Text.class);

    job.setOutputFormatClass(SequenceFileOutputFormat.class);
    job.setInputFormatClass(TextInputFormat.class);

    TextInputFormat.addInputPath(job, new Path("/lol"));
    SequenceFileOutputFormat.setOutputPath(job, new Path("/lolz"));

    // submit and wait for completion
    job.waitForCompletion(true);
   }

More Related Contents:

  1. Hive: Best way to do incremetal updates on a main table
  2. Assign ArrayList to File[] in java
  3. Recursively list files in Java
  4. Convert String to double in Java
  5. Fastest way to write huge data in text file Java
  6. How to list the files inside a JAR file?
  7. Move / Copy File Operations in Java
  8. log4j redirect stdout to DailyRollingFileAppender
  9. Java FileReader encoding issue
  10. Platform independent paths in Java
  11. How to get names of classes inside a jar file?
  12. Calling a mapreduce job from a simple java program
  13. What is the difference between Double.parseDouble(String) and Double.valueOf(String)?
  14. How to get the filename without the extension in Java?
  15. Difference between getExternalFilesDir and getExternalStorageDirectory()
  16. Counting the number of files in a directory using Java
  17. How to copy file inside jar to outside the jar?
  18. Make a File/Folder Hidden on Windows with Java
  19. How to use Sqoop in Java Program?
  20. Creating Runnable Jar with external files included
  21. Hadoop DistributedCache is deprecated – what is the preferred API?
  22. Iterate twice on values (MapReduce)
  23. Converting char[] to byte[]
  24. How do I delete files programmatically on Android?
  25. Converting from byte to int in Java
  26. How to break a file into pieces using Java?
  27. Converting some hexadecimal string to a decimal integer
  28. How to read from files with Files.lines(…).forEach(…)?
  29. Eclipse – Show generated class files
  30. Read directory inside JAR with InputStreamReader

Leave a Comment