How to use regex to include/exclude some input files in sc.textFile?

by

Looking at the accepted answer, it seems to use some form of glob syntax. It also reveals that the API is an exposure of Hadoop’s FileInputFormat.

Searching reveals that paths supplied to FileInputFormat‘s addInputPath or setInputPath “may represent a file, a directory, or, by using glob, a collection of files and directories”. Perhaps, SparkContext also uses those APIs to set the path.

The syntax of the glob includes:

  • * (match 0 or more character)
  • ? (match single character)
  • [ab] (character class)
  • [^ab] (negated character class)
  • [a-b] (character range)
  • {a,b} (alternation)
  • \c (escape character)

Following the example in the accepted answer, it is possible to write your path as:

sc.textFile("/user/Orders/2015072[7-9]*,/user/Orders/2015073[0-1]*")

It’s not clear how alternation syntax can be used here, since comma is used to delimit a list of paths (as shown above). According to zero323‘s comment, no escaping is necessary:

sc.textFile("/user/Orders/201507{2[7-9],3[0-1]}*")

More Related Contents:

  1. Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only
  2. Spark – load CSV file as DataFrame?
  3. How to define partitioning of DataFrame?
  4. Encoder error while trying to map dataframe row to updated row
  5. Case class equality in Apache Spark
  6. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  7. How can I change column types in Spark SQL’s DataFrame?
  8. how to make saveAsTextFile NOT split output into multiple file?
  9. Dropping a nested column from Spark DataFrame
  10. How to read records in JSON format from Kafka using Structured Streaming?
  11. Spark dataframe write method writing many small files
  12. Filling gaps in timeseries Spark
  13. Reading DataFrame from partitioned parquet file
  14. java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags
  15. How to use COGROUP for large datasets
  16. What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
  17. Parsing multiline records in Scala
  18. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  19. Create new Dataframe with empty/null field values
  20. DataFrame-ified zipWithIndex
  21. Spark: Add column to dataframe conditionally
  22. Explanation of fold method of spark RDD
  23. How to explode an array into multiple columns in Spark
  24. Spark2.1.0 incompatible Jackson versions 2.7.6
  25. How to sort an RDD in Scala Spark?
  26. Coalesce reduces parallelism of entire stage (spark)
  27. Difference between two rows in Spark dataframe
  28. Run a scala code jar appear NoSuchMethodError:scala.Predef$.refArrayOps
  29. Spark UDF with varargs
  30. How to transpose an RDD in Spark

Leave a Comment