Loading Data from a .txt file to Table Stored as ORC in Hive

by

LOAD DATA just copies the files to hive datafiles. Hive does not do any transformation while loading data into tables.

So, in this case the input file /home/user/test_details.txt needs to be in ORC format if you are loading it into an ORC table.

A possible workaround is to create a temporary table with STORED AS TEXT, then LOAD DATA into it, and then copy data from this table to the ORC table.

Here is an example:

CREATE TABLE test_details_txt( visit_id INT, store_id SMALLINT) STORED AS TEXTFILE;
CREATE TABLE test_details_orc( visit_id INT, store_id SMALLINT) STORED AS ORC;

-- Load into Text table
LOAD DATA LOCAL INPATH '/home/user/test_details.txt' INTO TABLE test_details_txt;

-- Copy to ORC table
INSERT INTO TABLE test_details_orc SELECT * FROM test_details_txt;

More Related Contents:

  1. What is the difference between partitioning and bucketing a table in Hive ?
  2. Difference between Hive internal tables and external tables?
  3. How to transpose/pivot data in hive?
  4. Sqoop import : composite primary key and textual primary key
  5. How to delete and update a record in Hive
  6. Hive unable to manually set number of reducers
  7. How to update table in Hive 0.13?
  8. how many mappers and reduces will get created for a partitoned table in hive
  9. Hive: Add partitions for existing folder structure
  10. How to load data to hive from HDFS without removing the source file?
  11. Create HIVE Table with multi character delimiter
  12. Job queue for Hive action in oozie
  13. Difference between Pig and Hive? Why have both? [closed]
  14. What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
  15. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?
  16. Hive Data Retrieval Queries: Difference between CLUSTER BY, ORDER BY, and SORT BY
  17. Begenner at spark Big data programming (spark code)
  18. Failed to locate the winutils binary in the hadoop binary path
  19. How does Hadoop Namenode failover process works?
  20. Hadoop java.io.IOException: Mkdirs failed to create /some/path
  21. Hive query performance for high cardinality field
  22. How to convert .txt file to Hadoop’s sequence file format
  23. Hadoop speculative task execution
  24. Is it better to have one large parquet file or lots of smaller parquet files?
  25. Easiest way to install Python dependencies on Spark executor nodes?
  26. Hadoop input split size vs block size
  27. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  28. Datanode process not running in Hadoop
  29. What is Google’s Dremel? How is it different from Mapreduce?
  30. Behavior of the parameter “mapred.min.split.size” in HDFS

Leave a Comment