hadoop No FileSystem for scheme: file

by

This is a typical case of the maven-assembly plugin breaking things.

Why this happened to us

Different JARs (hadoop-commons for LocalFileSystem, hadoop-hdfs for DistributedFileSystem) each contain a different file called org.apache.hadoop.fs.FileSystem in their META-INFO/services directory. This file lists the canonical classnames of the filesystem implementations they want to declare (This is called a Service Provider Interface implemented via java.util.ServiceLoader, see org.apache.hadoop.FileSystem#loadFileSystems).

When we use maven-assembly-plugin, it merges all our JARs into one, and all META-INFO/services/org.apache.hadoop.fs.FileSystem overwrite each-other. Only one of these files remains (the last one that was added). In this case, the FileSystem list from hadoop-commons overwrites the list from hadoop-hdfs, so DistributedFileSystem was no longer declared.

How we fixed it

After loading the Hadoop configuration, but just before doing anything FileSystem-related, we call this:

    hadoopConfig.set("fs.hdfs.impl", 
        org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
    );
    hadoopConfig.set("fs.file.impl",
        org.apache.hadoop.fs.LocalFileSystem.class.getName()
    );

Update: the correct fix

It has been brought to my attention by krookedking that there is a configuration-based way to make the maven-assembly use a merged version of all the FileSystem services declarations, check out his answer below.

More Related Contents:

  1. Fairly new to programming, trouble reading and writing to a binary file. [closed]
  2. File I/O Concept [closed]
  3. French and Chinese characters are not appearing correctly
  4. How to read all files in a folder from Java?
  5. How do I check if a file exists in Java?
  6. Easy way to write contents of a Java InputStream to an OutputStream
  7. Quickly read the last line of a text file?
  8. Use Cases and Examples of GoF Decorator Pattern for IO
  9. Do I need to close() both FileReader and BufferedReader?
  10. Create a new line in Java’s FileWriter
  11. Get an OutputStream into a String
  12. Printing my Mac’s serial number in java using Unix commands
  13. Oozie: Launch Map-Reduce from Oozie action?
  14. Hadoop cluster setup – java.net.ConnectException: Connection refused
  15. How to write data to two java.io.OutputStream objects at once?
  16. Reading file as single record in hadoop
  17. How to convert .txt file to Hadoop’s sequence file format
  18. java read file from network device
  19. Sending the same but modifed object over ObjectOutputStream
  20. read complete file without using loop in java
  21. Java Scanner doesn’t wait for user input [duplicate]
  22. MultipleOutputFormat in hadoop
  23. out of Memory Error in Hadoop
  24. How can I force Spark to execute code?
  25. Change File Split size in Hadoop
  26. How to list all files in a directory and its subdirectories in hadoop hdfs
  27. hadoop java.net.URISyntaxException: Relative path in absolute URI: rsrc:hbase-common-0.98.1-hadoop2.jar
  28. Resetting Standard output Stream
  29. java file input with rewind()/reset() capability
  30. Will using longs instead of ints benefit in 64bit java

Leave a Comment