How to process multi line input records in Spark

by

If the multi-line data has a defined record separator, you could use the hadoop support for multi-line records, providing the separator through a hadoop.Configuration object:

Something like this should do:

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.io.{LongWritable, Text}
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat
val conf = new Configuration
conf.set("textinputformat.record.delimiter", "id:")
val dataset = sc.newAPIHadoopFile("/path/to/data", classOf[TextInputFormat], classOf[LongWritable], classOf[Text], conf)
val data = dataset.map(x=>x._2.toString)

This will provide you with an RDD[String] where each element corresponds to a record. Afterwards you need to parse each record following your application requirements.

More Related Contents:

  1. Write single CSV file using spark-csv
  2. (Why) do we need to call cache or persist on a RDD
  3. How to load local file in sc.textFile, instead of HDFS
  4. Automatically and Elegantly flatten DataFrame in Spark SQL
  5. How do I skip a header from CSV files in Spark?
  6. Spark extracting values from a Row
  7. Stratified sampling in Spark
  8. How do I convert csv file to rdd
  9. How to pass -D parameter or environment variable to Spark job?
  10. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  11. Append a column to Data Frame in Apache Spark 1.3
  12. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  13. Processing multiple files as independent RDD’s in parallel
  14. MatchError while accessing vector column in Spark 2.0
  15. How to save DataFrame directly to Hive?
  16. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  17. Spark / Scala: forward fill with last observation
  18. How to convert unix timestamp to date in Spark
  19. Function returns an empty List in Spark
  20. Apache Spark: Get number of records per partition
  21. Derive multiple columns from a single column in a Spark DataFrame
  22. How to save a spark DataFrame as csv on disk?
  23. Exploding nested Struct in Spark dataframe
  24. Spark: long delay between jobs
  25. How do I iterate RDD’s in apache spark (scala)
  26. How to get keys and values from MapType column in SparkSQL DataFrame
  27. Addition of two RDD[mllib.linalg.Vector]’s
  28. Add Number of days column to Date Column in same dataframe for Spark Scala App
  29. How to create a Dataset of Maps?
  30. Spark ML VectorAssembler returns strange output

Leave a Comment