Pig Latin: Load multiple files from a date range (part of the directory structure)

by

As zjffdu said, the path expansion is done by the shell. One common way to solve your problem is to simply use Pig parameters (which is a good way to make your script more resuable anyway):

shell:

pig -f script.pig -param input=/user/training/test/{20100810..20100812}

script.pig:

temp = LOAD '$input' USING SomeLoader() AS (...);

More Related Contents:

  1. Difference between Pig and Hive? Why have both? [closed]
  2. Merging multiple files into one within Hadoop
  3. How does Hadoop process records split across block boundaries?
  4. Failed to locate the winutils binary in the hadoop binary path
  5. merge output files after reduce phase
  6. What is the difference between partitioning and bucketing a table in Hive ?
  7. Setting the number of map tasks and reduce tasks
  8. Difference between Hive internal tables and external tables?
  9. How to transpose/pivot data in hive?
  10. How to open/stream .zip files through Spark?
  11. Name node is in safe mode. Not able to leave
  12. How to fix corrupt HDFS FIles
  13. Is it better to use the mapred or the mapreduce package to create a Hadoop Job?
  14. Sqoop import : composite primary key and textual primary key
  15. When do reduce tasks start in Hadoop?
  16. How to delete and update a record in Hive
  17. Hadoop java.io.IOException: Mkdirs failed to create /some/path
  18. How can I access S3/S3n from a local Hadoop 2.6 installation?
  19. Hadoop speculative task execution
  20. Hive unable to manually set number of reducers
  21. How to update table in Hive 0.13?
  22. Easiest way to install Python dependencies on Spark executor nodes?
  23. Parallel Algorithms for Generating Prime Numbers (possibly using Hadoop’s map reduce)
  24. Spark iterate HDFS directory
  25. Loading Data from a .txt file to Table Stored as ORC in Hive
  26. Hadoop: …be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation
  27. Cannot Read a file from HDFS using Spark
  28. Hadoop WordCount example stuck at map 100% reduce 0%
  29. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?
  30. hadoop: difference between 0 reducer and identity reducer?

Leave a Comment