How do I Combine or Merge Small ORC files into Larger ORC file?

by

You do not need to re-invent the wheel.

ALTER TABLE table_name [PARTITION partition_spec] CONCATENATE can be used to merge small ORC files into a larger file since Hive 0.14.0. The merge happens at the stripe level, which avoids decompressing and decoding the data. It works fast. I’d suggest to create an external table partitioned by day (partitions are directories), then merge them all specifying PARTITION (day_column) as a partition spec.

See here: LanguageManual+ORC

More Related Contents:

  1. Hive: Best way to do incremetal updates on a main table
  2. Read whole text files from a compression in Spark
  3. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  4. Write a file in hdfs with Java
  5. How to convert .txt file to Hadoop’s sequence file format
  6. How to list all files in a directory and its subdirectories in hadoop hdfs
  7. How to programmatically close a JFrame
  8. Why is spawning threads in Java EE container discouraged?
  9. What is the best way to implement constants in Java? [closed]
  10. What is the best approach for using an Enum as a singleton in Java?
  11. What is the actual use of Class.forName(“oracle.jdbc.driver.OracleDriver”) while connecting to a database?
  12. Expected BEGIN_ARRAY but was BEGIN_OBJECT at line 1 column 2
  13. How to change JDK version for an Eclipse project
  14. Optimization by Java Compiler
  15. How to create an FXML file for an already created new component in java than add it to scene builder?
  16. How can I return the difference between two lists?
  17. Java: Find .txt files in specified folder
  18. What is “compiler compliance level” in Eclipse?
  19. Capture SIGINT in Java
  20. What is the time complexity performance of HashSet.contains() in Java?
  21. How to connect Android with MySQL using Mysql JDBC driver [duplicate]
  22. Environment Variable with Maven
  23. JDBC ResultSet get columns with table alias
  24. Android SQLite and huge data sets
  25. Setting up enviromental variables in Windows 10 to use java and javac
  26. java.io.InvalidClassException: local class incompatible:
  27. How do I unit test jdbc code in java? [closed]
  28. How can I remove punctuation from input text in Java?
  29. What are the Java regular expressions for matching IPv4 and IPv6 strings?
  30. convert little Endian file into big Endian

Leave a Comment