Spark iterate HDFS directory

by

You can use org.apache.hadoop.fs.FileSystem. Specifically, FileSystem.listFiles([path], true)

And with Spark…

FileSystem.get(sc.hadoopConfiguration).listFiles(..., true)

Edit

It’s worth noting that good practice is to get the FileSystem that is associated with the Path‘s scheme.

path.getFileSystem(sc.hadoopConfiguration).listFiles(path, true)

More Related Contents:

  1. Spark on yarn concept understanding
  2. Begenner at spark Big data programming (spark code)
  3. Write to multiple outputs by key Spark – one Spark job
  4. Spark – load CSV file as DataFrame?
  5. How does Hadoop process records split across block boundaries?
  6. While writing to hdfs path getting error java.io.IOException: Failed to rename
  7. Read whole text files from a compression in Spark
  8. How does Hadoop Namenode failover process works?
  9. How to open/stream .zip files through Spark?
  10. Name node is in safe mode. Not able to leave
  11. How to fix corrupt HDFS FIles
  12. Hadoop java.io.IOException: Mkdirs failed to create /some/path
  13. Pyspark: get list of files/directories on HDFS path
  14. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  15. How to access s3a:// files from Apache Spark?
  16. Is it better to have one large parquet file or lots of smaller parquet files?
  17. Apache Spark: The number of cores vs. the number of executors
  18. Apache Hadoop Yarn – Underutilization of cores
  19. Where are logs in Spark on YARN?
  20. Namenode not getting started
  21. Easiest way to install Python dependencies on Spark executor nodes?
  22. Hive: Add partitions for existing folder structure
  23. Default Namenode port of HDFS is 50070.But I have come across at some places 8020 or 9000 [closed]
  24. How does Hadoop perform input splits?
  25. Hadoop: …be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation
  26. Default number of reducers
  27. Difference between HBase and Hadoop/HDFS
  28. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  29. Cannot Read a file from HDFS using Spark
  30. Behavior of the parameter “mapred.min.split.size” in HDFS

Leave a Comment