Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:

by

By default, this data is on the DBFS, and your code need to understand how to access it. Python doesn’t know about it – that’s why it’s failing.

But there is a workaround – DBFS is mounted to the nodes at /dbfs, so you just need to append it to your file name: instead of /user/delta_test/_delta_log/00000000000000000000.json, use /dbfs/user/delta_test/_delta_log/00000000000000000000.json

update: on community edition, in DBR 7+, this mount is disabled. The workaround would be to use dbutils.fs.cp command to copy file from DBFS to local directory, like, /tmp, or /var/tmp, and then read from it:

dbutils.fs.cp("/file_on_dbfs", "file:///tmp/local_file")

please note that if you don’t specify URI schema, then the file by default is referring DBFS, and to refer the local file you need to use file:// prefix (see docs).

More Related Contents:

  1. How can I access python variable in Spark SQL?
  2. Finding duplicates from large data set using Apache Spark
  3. Using a column value as a parameter to a spark DataFrame function
  4. Find maximum row per group in Spark DataFrame
  5. How do I split an RDD into two or more RDDs?
  6. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  7. Split Spark Dataframe string column into multiple columns
  8. How to access element of a VectorUDT column in a Spark DataFrame?
  9. Python Spark Cumulative Sum by Group Using DataFrame
  10. How to split a list to multiple columns in Pyspark?
  11. Link Spark with iPython Notebook
  12. How to save/insert each DStream into a permanent table
  13. Spark load data and add filename as dataframe column
  14. PySpark: how to resample frequencies
  15. Efficient pyspark join
  16. Explode array data into rows in spark [duplicate]
  17. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  18. PySpark – get row number for each row in a group
  19. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  20. Save ML model for future usage
  21. Apache spark dealing with case statements
  22. Keep only duplicates from a DataFrame regarding some field
  23. pyspark: count distinct over a window
  24. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  25. How to exclude multiple columns in Spark dataframe in Python
  26. Spark + s3 – error – java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
  27. Stratified sampling with pyspark
  28. Get CSV to Spark dataframe
  29. How to turn off scientific notation in pyspark?
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment