Pyspark: get list of files/directories on HDFS path

by

Using JVM gateway maybe is not so elegant, but in some cases the code below could be helpful:

URI           = sc._gateway.jvm.java.net.URI
Path          = sc._gateway.jvm.org.apache.hadoop.fs.Path
FileSystem    = sc._gateway.jvm.org.apache.hadoop.fs.FileSystem
Configuration = sc._gateway.jvm.org.apache.hadoop.conf.Configuration


fs = FileSystem.get(URI("hdfs://somehost:8020"), Configuration())

status = fs.listStatus(Path('/some_dir/yet_another_one_dir/'))

for fileStatus in status:
    print(fileStatus.getPath())

More Related Contents:

  1. Begenner at spark Big data programming (spark code)
  2. How to turn off INFO logging in Spark?
  3. How to open/stream .zip files through Spark?
  4. Spark on yarn concept understanding
  5. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  6. How to access s3a:// files from Apache Spark?
  7. Is it better to have one large parquet file or lots of smaller parquet files?
  8. Apache Spark: The number of cores vs. the number of executors
  9. Apache Hadoop Yarn – Underutilization of cores
  10. Where are logs in Spark on YARN?
  11. Easiest way to install Python dependencies on Spark executor nodes?
  12. Spark iterate HDFS directory
  13. Cannot Read a file from HDFS using Spark
  14. Information about big data and hadoop [closed]
  15. Spark: Inconsistent performance number in scaling number of cores
  16. Spark Dataframe validating column names for parquet writes
  17. TypeError: Column is not iterable – How to iterate over ArrayType()?
  18. Chaining multiple MapReduce jobs in Hadoop
  19. How to run multiple jobs in one Sparkcontext from separate threads in PySpark?
  20. Why does Spark think this is a cross / Cartesian join
  21. Pyspark: Replacing value in a column by searching a dictionary
  22. Pivot String column on Pyspark Dataframe
  23. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  24. how many mappers and reduces will get created for a partitoned table in hive
  25. Reduce a key-value pair into a key-list pair with Apache Spark
  26. PySpark in iPython notebook raises Py4JJavaError when using count() and first()
  27. How to load data to hive from HDFS without removing the source file?
  28. pyspark: Efficiently have partitionBy write to same number of total partitions as original table
  29. Hadoop WordCount example stuck at map 100% reduce 0%
  30. Spark ALS predictAll returns empty

Leave a Comment