Spark 2.0.x dump a csv file from a dataframe containing one array of type string

by

The reason why you are getting this error is that csv file format doesn’t support array types, you’ll need to express it as a string to be able to save.

Try the following :

import org.apache.spark.sql.functions._

val stringify = udf((vs: Seq[String]) => vs match {
  case null => null
  case _    => s"""[${vs.mkString(",")}]"""
})

df.withColumn("ArrayOfString", stringify($"ArrayOfString")).write.csv(...)

or

import org.apache.spark.sql.Column

def stringify(c: Column) = concat(lit("["), concat_ws(",", c), lit("]"))

df.withColumn("ArrayOfString", stringify($"ArrayOfString")).write.csv(...)

More Related Contents:

  1. Spark Error:expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)
  2. Access Array column in Spark
  3. Getting “System.Collections.Generic.List`1[System.String]” in CSV File export when data is okay on screen
  4. Asynchronous access to an array in Firebase
  5. Capturing output of find . -print0 into a bash array
  6. How do I slice an array in Excel VBA?
  7. MongoDB: How do I update a single subelement in an array, referenced by the index within the array?
  8. Unnest multiple arrays in parallel
  9. Comparing two arrays & get the values which are not common
  10. Algorithm to find k smallest numbers in array of n items
  11. Can PostgreSQL have a uniqueness constraint on array elements?
  12. How to pass allocatable arrays to subroutines in Fortran
  13. Shorthand to test if an object exists in an array for Swift?
  14. Automatic array allocation upon assignment in Fortran
  15. Flatten [Any] Array Swift
  16. Given an array, find out the next smaller element for each element
  17. Algorithm to rotate an array in linear time
  18. How to pass Scala array into Scala vararg method?
  19. Using jsonb_set() for updating specific jsonb array value
  20. Importing data from multiple csv files in D3
  21. In-place array reordering?
  22. Subscript out of range error with an array – no idea why?
  23. In Fortran 90, do array dimensions have to be declared beforehand?
  24. Translating C function with args to MIPS: loop over an int array and count negatives
  25. swift: modifying arrays inside dictionaries
  26. find pair of numbers in array that add to given sum
  27. Underscore.js groupBy multiple values
  28. How can I store function pointers in an array? [duplicate]
  29. Is there any way to allocate a standard Rust array directly on the heap, skipping the stack entirely?
  30. Swift: Binary search for standard array?

Leave a Comment