Get CSV to Spark dataframe

by

With more recent versions of Spark (as of, I believe, 1.4) this has become a lot easier. The expression sqlContext.read gives you a DataFrameReader instance, with a .csv() method:

df = sqlContext.read.csv("/path/to/your.csv")

Note that you can also indicate that the csv file has a header by adding the keyword argument header=True to the .csv() call. A handful of other options are available, and described in the link above.

More Related Contents:

  1. How to melt Spark DataFrame?
  2. Using a column value as a parameter to a spark DataFrame function
  3. Unpivot in spark-sql/pyspark
  4. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  5. Split Spark Dataframe string column into multiple columns
  6. Avoid performance impact of a single partition mode in Spark window functions
  7. How to check if spark dataframe is empty?
  8. TypeError: Column is not iterable – How to iterate over ArrayType()?
  9. How to split a list to multiple columns in Pyspark?
  10. pyspark dataframe filter or include based on list
  11. How to fix ‘TypeError: an integer is required (got type bytes)’ error when trying to run pyspark after installing spark 2.4.4
  12. Pyspark : forward fill with last observation for a DataFrame
  13. Adding a group count column to a PySpark dataframe
  14. Efficient pyspark join
  15. Multiple Spark applications with HiveContext
  16. How to loop through each row of dataFrame in pyspark
  17. Fill in null with previously known good value with pyspark
  18. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  19. PySpark – get row number for each row in a group
  20. Save ML model for future usage
  21. Pyspark: Pass multiple columns in UDF
  22. Keep only duplicates from a DataFrame regarding some field
  23. reading json file in pyspark
  24. How can I access python variable in Spark SQL?
  25. PySpark: How to fillna values in dataframe for specific columns?
  26. pyspark: rolling average using timeseries data
  27. Spark ALS predictAll returns empty
  28. Why is the fold action necessary in Spark?
  29. How to add a SparkListener from pySpark in Python?
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment