Cannot Read a file from HDFS using Spark

by

Here is the solution

sc.textFile("hdfs://nn1home:8020/input/war-and-peace.txt")

How did I find out nn1home:8020?

Just search for the file core-site.xml and look for xml element fs.defaultFS

More Related Contents:

  1. Begenner at spark Big data programming (spark code)
  2. How to open/stream .zip files through Spark?
  3. Spark on yarn concept understanding
  4. Pyspark: get list of files/directories on HDFS path
  5. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  6. How to access s3a:// files from Apache Spark?
  7. Is it better to have one large parquet file or lots of smaller parquet files?
  8. Apache Spark: The number of cores vs. the number of executors
  9. Apache Hadoop Yarn – Underutilization of cores
  10. Where are logs in Spark on YARN?
  11. Easiest way to install Python dependencies on Spark executor nodes?
  12. Spark iterate HDFS directory
  13. How to turn off INFO logging in Spark?
  14. How does Hadoop Namenode failover process works?
  15. How can PySpark be called in debug mode?
  16. Sqoop import : composite primary key and textual primary key
  17. When do reduce tasks start in Hadoop?
  18. Hadoop java.io.IOException: Mkdirs failed to create /some/path
  19. Set hadoop system user for client embedded in Java webapp
  20. How can I access S3/S3n from a local Hadoop 2.6 installation?
  21. OOZIE: properties defined in file referenced in global job-xml not visible in workflow.xml
  22. How to get the input file name in the mapper in a Hadoop program?
  23. hadoop map reduce secondary sorting
  24. How can I force Spark to execute code?
  25. Parallel Algorithms for Generating Prime Numbers (possibly using Hadoop’s map reduce)
  26. Default Namenode port of HDFS is 50070.But I have come across at some places 8020 or 9000 [closed]
  27. Spark Scala list folders in directory
  28. Hadoop input split size vs block size
  29. Hadoop WordCount example stuck at map 100% reduce 0%
  30. hadoop: difference between 0 reducer and identity reducer?

Leave a Comment