Iterate twice on values (MapReduce)

by

Unfortunately this is not possible without caching the values as in Andreas_D’s answer.

Even using the new API, where the Reducer receives an Iterable rather than an Iterator, you cannot iterate twice. It’s very tempting to try something like:

for (IntWritable value : values) {
    // first loop
}

for (IntWritable value : values) {
    // second loop
}

But this won’t actually work. The Iterator you receive from that Iterable‘s iterator() method is special. The values may not all be in memory; Hadoop may be streaming them from disk. They aren’t really backed by a Collection, so it’s nontrivial to allow multiple iterations.

You can see this for yourself in the Reducer and ReduceContext code.

Caching the values in a Collection of some sort may be the easiest answer, but you can easily blow the heap if you are operating on large datasets. If you can give us more specifics on your problem, we may be able to help you find a solution that doesn’t involve multiple iterations.

More Related Contents:

  1. Hadoop “Unable to load native-hadoop library for your platform” warning
  2. Adding elements to a collection during iteration
  3. How to convert an Iterator to a Stream?
  4. hadoop No FileSystem for scheme: file
  5. Write a file in hdfs with Java
  6. Performance of traditional for loop vs Iterator/foreach in Java
  7. Should I call ugi.checkTGTAndReloginFromKeytab() before every action on hadoop?
  8. Can we write our own iterator in Java?
  9. Oozie: Launch Map-Reduce from Oozie action?
  10. Difference between Iterator and Listiterator?
  11. Hadoop cluster setup – java.net.ConnectException: Connection refused
  12. Hadoop Word count: receive the total number of words that start with the letter “c”
  13. Reading file as single record in hadoop
  14. How Iterator’s remove method actually remove an object
  15. How to convert .txt file to Hadoop’s sequence file format
  16. Accessing a mapper’s counter from a reducer
  17. How do I build/run this simple Mahout program without getting exceptions?
  18. MultipleOutputFormat in hadoop
  19. How can I implement the Iterable interface?
  20. out of Memory Error in Hadoop
  21. How can I force Spark to execute code?
  22. Generator functions equivalent in Java
  23. Change File Split size in Hadoop
  24. How to list all files in a directory and its subdirectories in hadoop hdfs
  25. Java: why can’t iterate over an iterator?
  26. How to run a jar file in hadoop?
  27. hadoop java.net.URISyntaxException: Relative path in absolute URI: rsrc:hbase-common-0.98.1-hadoop2.jar
  28. What is the best way to get the count/length/size of an iterator?
  29. Iterating over and deleting from Hashtable in Java
  30. Provide an iterator over the contents of two lists simultaneously?

Leave a Comment