out of Memory Error in Hadoop

by

For anyone using RPM or DEB packages, the documentation and common advice is misleading. These packages install hadoop configuration files into /etc/hadoop. These will take priority over other settings.

The /etc/hadoop/hadoop-env.sh sets the maximum java heap memory for Hadoop, by Default it is:

   export HADOOP_CLIENT_OPTS="-Xmx128m $HADOOP_CLIENT_OPTS"

This Xmx setting is too low, simply change it to this and rerun

   export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"

More Related Contents:

  1. Hadoop “Unable to load native-hadoop library for your platform” warning
  2. Hive: Best way to do incremetal updates on a main table
  3. Read whole text files from a compression in Spark
  4. hadoop No FileSystem for scheme: file
  5. Write a file in hdfs with Java
  6. Should I call ugi.checkTGTAndReloginFromKeytab() before every action on hadoop?
  7. Calling a mapreduce job from a simple java program
  8. How to use Sqoop in Java Program?
  9. Oozie: Launch Map-Reduce from Oozie action?
  10. Hadoop DistributedCache is deprecated – what is the preferred API?
  11. Hadoop cluster setup – java.net.ConnectException: Connection refused
  12. Hadoop Word count: receive the total number of words that start with the letter “c”
  13. Reading file as single record in hadoop
  14. How to convert .txt file to Hadoop’s sequence file format
  15. Exception in thread “main” java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  16. Iterate twice on values (MapReduce)
  17. Accessing a mapper’s counter from a reducer
  18. NotSerializableException on anonymous class
  19. How do I build/run this simple Mahout program without getting exceptions?
  20. MultipleOutputFormat in hadoop
  21. How can I force Spark to execute code?
  22. Change File Split size in Hadoop
  23. How to list all files in a directory and its subdirectories in hadoop hdfs
  24. How to run a jar file in hadoop?
  25. hadoop java.net.URISyntaxException: Relative path in absolute URI: rsrc:hbase-common-0.98.1-hadoop2.jar
  26. Exception “: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length” from java
  27. How to read an external properties file in Maven
  28. Java Inheritance – instance variables overriding
  29. OpenCV Template Matching example in Android
  30. Stream Filter of 1 list based on another list

Leave a Comment