Spark Scala list folders in directory

by

We are using hadoop 1.4 and it doesn’t have listFiles method so we use listStatus to get directories. It doesn’t have recursive option but it is easy to manage recursive lookup.

val fs = FileSystem.get(new Configuration())
val status = fs.listStatus(new Path(YOUR_HDFS_PATH))
status.foreach(x=> println(x.getPath))

More Related Contents:

  1. Write to multiple outputs by key Spark – one Spark job
  2. Spark – load CSV file as DataFrame?
  3. How to get ID of a map task in Spark?
  4. Apache Spark: Get number of records per partition
  5. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  6. Spark: long delay between jobs
  7. (Why) do we need to call cache or persist on a RDD
  8. Automatically and Elegantly flatten DataFrame in Spark SQL
  9. How do I skip a header from CSV files in Spark?
  10. Spark extracting values from a Row
  11. How to pass -D parameter or environment variable to Spark job?
  12. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  13. Append a column to Data Frame in Apache Spark 1.3
  14. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  15. Processing multiple files as independent RDD’s in parallel
  16. MatchError while accessing vector column in Spark 2.0
  17. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  18. Spark / Scala: forward fill with last observation
  19. How to convert unix timestamp to date in Spark
  20. Derive multiple columns from a single column in a Spark DataFrame
  21. How to save a spark DataFrame as csv on disk?
  22. How to process multi line input records in Spark
  23. How to sort by column in descending order in Spark SQL?
  24. How to get keys and values from MapType column in SparkSQL DataFrame
  25. Addition of two RDD[mllib.linalg.Vector]’s
  26. How to sort an RDD in Scala Spark?
  27. Difference between two rows in Spark dataframe
  28. Run a scala code jar appear NoSuchMethodError:scala.Predef$.refArrayOps
  29. Spark UDF with varargs
  30. How to transpose an RDD in Spark

Leave a Comment