How to read multiple text files into a single RDD?

by

You can specify whole directories, use wildcards and even CSV of directories and wildcards. E.g.:

sc.textFile("/my/dir1,/my/paths/part-00[0-5]*,/another/dir,/a/specific/file")

As Nick Chammas points out this is an exposure of Hadoop’s FileInputFormat and therefore this also works with Hadoop (and Scalding).

More Related Contents:

  1. How to melt Spark DataFrame?
  2. Spark SQL: apply aggregate functions to a list of columns
  3. Unpivot in spark-sql/pyspark
  4. Spark SQL replacement for MySQL’s GROUP_CONCAT aggregate function
  5. Spark Dataframe validating column names for parquet writes
  6. How to optimize partitioning when migrating data from JDBC source?
  7. Spark: subtract two DataFrames
  8. TypeError: Column is not iterable – How to iterate over ArrayType()?
  9. How to overwrite the output directory in spark
  10. Is groupByKey ever preferred over reduceByKey
  11. Pyspark : forward fill with last observation for a DataFrame
  12. spark.sql.crossJoin.enabled for Spark 2.x
  13. Why do I have to explicitly tell Spark what to cache?
  14. multiple conditions for filter in spark data frames
  15. Why does Spark think this is a cross / Cartesian join
  16. Why does sortBy transformation trigger a Spark job?
  17. How to pass whole Row to UDF – Spark DataFrame filter
  18. Spark SQL broadcast hash join
  19. Rename more than one column using withColumnRenamed
  20. What is the difference between Apache Spark SQLContext vs HiveContext?
  21. pyspark: Efficiently have partitionBy write to same number of total partitions as original table
  22. Spark lists all leaf node even in partitioned data
  23. Spark gives a StackOverflowError when training using ALS
  24. Mac spark-shell Error initializing SparkContext
  25. What are broadcast variables? What problems do they solve?
  26. Why spark application fail with “executor.CoarseGrainedExecutorBackend: Driver Disassociated”?
  27. What does “Correlated scalar subqueries must be Aggregated” mean?
  28. Why is the fold action necessary in Spark?
  29. How to add a SparkListener from pySpark in Python?
  30. What will spark do if I don’t have enough memory?

Leave a Comment