How do I skip a header from CSV files in Spark?

by 
data = sc.textFile('path_to_data')
header = data.first() #extract header
data = data.filter(row => row != header)   #filter out header

More Related Contents:

  1. Write single CSV file using spark-csv
  2. Specifying the filename when saving a DataFrame as a CSV [duplicate]
  3. How to bucket the range of values from a column and count how many values fall into each interval in scala?
  4. How to store custom objects in Dataset?
  5. How do I detect if a Spark DataFrame has a column
  6. How to define schema for custom type in Spark SQL?
  7. How to avoid duplicate columns after join?
  8. Flattening Rows in Spark
  9. Spark unionAll multiple dataframes
  10. Spark performance for Scala vs Python
  11. How to zip two (or more) DataFrame in Spark
  12. How to force DataFrame evaluation in Spark
  13. How to write spark streaming DF to Kafka topic
  14. Stackoverflow due to long RDD Lineage
  15. call of distinct and map together throws NPE in spark library
  16. How to get path to the uploaded file
  17. how to filter out a null value from spark dataframe
  18. Is there a reason not to use SparkContext.getOrCreate when writing a spark job?
  19. What are the various join types in Spark?
  20. Provide schema while reading csv file as a dataframe in Scala Spark
  21. Replace missing values with mean – Spark Dataframe
  22. How to calculate the size of dataframe in bytes in Spark?
  23. Why does partition parameter of SparkContext.textFile not take effect?
  24. Spark Scala list folders in directory
  25. Exception in thread “main” java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)
  26. Spark-Monotonically increasing id not working as expected in dataframe?
  27. How to remove parentheses around records when saveAsTextFile on RDD[(String, Int)]?
  28. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  29. How to create a Dataset of Maps?
  30. Spark ML VectorAssembler returns strange output

Leave a Comment