Change File Split size in Hadoop

by

The parameter mapred.max.split.size which can be set per job individually is what you looking for. Don’t change dfs.block.size because this is global for HDFS and can lead to problems.

More Related Contents:

  1. Calling a mapreduce job from a simple java program
  2. Oozie: Launch Map-Reduce from Oozie action?
  3. Hadoop DistributedCache is deprecated – what is the preferred API?
  4. Hadoop Word count: receive the total number of words that start with the letter “c”
  5. Reading file as single record in hadoop
  6. MultipleOutputFormat in hadoop
  7. hadoop java.net.URISyntaxException: Relative path in absolute URI: rsrc:hbase-common-0.98.1-hadoop2.jar
  8. Hadoop “Unable to load native-hadoop library for your platform” warning
  9. Hive: Best way to do incremetal updates on a main table
  10. Read whole text files from a compression in Spark
  11. hadoop No FileSystem for scheme: file
  12. Write a file in hdfs with Java
  13. Should I call ugi.checkTGTAndReloginFromKeytab() before every action on hadoop?
  14. What determines Kafka consumer offset?
  15. How to use Sqoop in Java Program?
  16. Hadoop cluster setup – java.net.ConnectException: Connection refused
  17. How to convert .txt file to Hadoop’s sequence file format
  18. Exception in thread “main” java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  19. Iterate twice on values (MapReduce)
  20. Accessing a mapper’s counter from a reducer
  21. NotSerializableException on anonymous class
  22. How do I build/run this simple Mahout program without getting exceptions?
  23. Java8: HashMap to HashMap using Stream / Map-Reduce / Collector
  24. Is gzip format supported in Spark?
  25. Change output filename prefix for DataFrame.write()
  26. out of Memory Error in Hadoop
  27. How can I force Spark to execute code?
  28. How to list all files in a directory and its subdirectories in hadoop hdfs
  29. How to run a jar file in hadoop?
  30. Exception “: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length” from java

Leave a Comment