Spark: Reading files using different delimiter than new line

by

You can use textinputformat.record.delimiter to set the delimiter for TextInputFormat, E.g.,

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.mapreduce.Job
import org.apache.hadoop.io.{LongWritable, Text}
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat

val conf = new Configuration(sc.hadoopConfiguration)
conf.set("textinputformat.record.delimiter", "X")
val input = sc.newAPIHadoopFile("file_path", classOf[TextInputFormat], classOf[LongWritable], classOf[Text], conf)
val lines = input.map { case (_, text) => text.toString}
println(lines.collect)

For example, my input is a file containing one line aXbXcXd. The above code will output 

Array(a, b, c, d)

More Related Contents:

  1. How to melt Spark DataFrame?
  2. How to stop INFO messages displaying on spark console?
  3. DataFrame join optimization – Broadcast Hash Join
  4. Overwrite specific partitions in spark dataframe write method
  5. Convert date from String to Date format in Dataframes
  6. How to check if spark dataframe is empty?
  7. What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?
  8. Spark Transformation – Why is it lazy and what is the advantage?
  9. What is the difference between spark checkpoint and persist to a disk
  10. How to fix ‘TypeError: an integer is required (got type bytes)’ error when trying to run pyspark after installing spark 2.4.4
  11. Pyspark : forward fill with last observation for a DataFrame
  12. Adding a group count column to a PySpark dataframe
  13. Multiple Spark applications with HiveContext
  14. NoClassDefFoundError com.apache.hadoop.fs.FSDataInputStream when execute spark-shell
  15. Spark: Best practice for retrieving big data from RDD to local machine
  16. Spark-Obtaining file name in RDDs
  17. How do I add an persistent column of row ids to Spark DataFrame?
  18. How does Spark aggregate function – aggregateByKey work?
  19. How to pass whole Row to UDF – Spark DataFrame filter
  20. Rename more than one column using withColumnRenamed
  21. Convert null values to empty array in Spark DataFrame
  22. DataFrame partitionBy to a single Parquet file (per partition)
  23. Temp table caching with spark-sql
  24. reduce result datasets into single dataset
  25. How to exclude multiple columns in Spark dataframe in Python
  26. Why does Spark job fail with “too many open files”?
  27. How does Distinct() function work in Spark?
  28. Stratified sampling with pyspark
  29. Get CSV to Spark dataframe
  30. How to turn off scientific notation in pyspark?

Leave a Comment