How to load local file in sc.textFile, instead of HDFS

by

Try explicitly specify sc.textFile("file:///path to the file/"). The error occurs when Hadoop environment is set.

SparkContext.textFile internally calls org.apache.hadoop.mapred.FileInputFormat.getSplits, which in turn uses org.apache.hadoop.fs.getDefaultUri if schema is absent. This method reads “fs.defaultFS” parameter of Hadoop conf. If you set HADOOP_CONF_DIR environment variable, the parameter is usually set as “hdfs://…”; otherwise “file://”.

More Related Contents:

  1. How MapReduce work in Apache Spark and Scala?
  2. Write to multiple outputs by key Spark – one Spark job
  3. How does HashPartitioner work?
  4. Spark 2.0 Dataset vs DataFrame
  5. Automatically and Elegantly flatten DataFrame in Spark SQL
  6. How do I skip a header from CSV files in Spark?
  7. Spark extracting values from a Row
  8. Stratified sampling in Spark
  9. How do I convert csv file to rdd
  10. How to pass -D parameter or environment variable to Spark job?
  11. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  12. Append a column to Data Frame in Apache Spark 1.3
  13. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  14. Processing multiple files as independent RDD’s in parallel
  15. MatchError while accessing vector column in Spark 2.0
  16. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  17. Spark / Scala: forward fill with last observation
  18. How to convert unix timestamp to date in Spark
  19. Apache Spark: Get number of records per partition
  20. Derive multiple columns from a single column in a Spark DataFrame
  21. How to save a spark DataFrame as csv on disk?
  22. How to process multi line input records in Spark
  23. How to sort by column in descending order in Spark SQL?
  24. How to get keys and values from MapType column in SparkSQL DataFrame
  25. Spark : how to run spark file from spark shell
  26. Spark Dataframe Random UUID changes after every transformation/action
  27. How to set hadoop configuration values from pyspark
  28. Spark DataFrames when udf functions do not accept large enough input variables
  29. Filter spark DataFrame on string contains
  30. Spark textFile vs wholeTextFiles

Leave a Comment