Generate a Spark StructType / Schema from a case class

by

You can do it the same way SQLContext.createDataFrame does it:

import org.apache.spark.sql.catalyst.ScalaReflection
val schema = ScalaReflection.schemaFor[TestCase].dataType.asInstanceOf[StructType]

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Using a column value as a parameter to a spark DataFrame function
  3. Find maximum row per group in Spark DataFrame
  4. Split Spark Dataframe string column into multiple columns
  5. While writing to hdfs path getting error java.io.IOException: Failed to rename
  6. Multiple Aggregate operations on the same column of a spark dataframe
  7. Partitioning in spark while reading from RDBMS via JDBC
  8. How to control partition size in Spark SQL
  9. How to import multiple csv files in a single load?
  10. How to access element of a VectorUDT column in a Spark DataFrame?
  11. How to split a list to multiple columns in Pyspark?
  12. pyspark dataframe filter or include based on list
  13. How to save/insert each DStream into a permanent table
  14. Use collect_list and collect_set in Spark SQL
  15. Spark load data and add filename as dataframe column
  16. What is the maximum size for a broadcast object in Spark?
  17. PySpark: how to resample frequencies
  18. How does createOrReplaceTempView work in Spark?
  19. Fill in null with previously known good value with pyspark
  20. PySpark – get row number for each row in a group
  21. Keep only duplicates from a DataFrame regarding some field
  22. pyspark: count distinct over a window
  23. Using windowing functions in Spark
  24. How can I access python variable in Spark SQL?
  25. PySpark: How to fillna values in dataframe for specific columns?
  26. How to get Kafka offsets for structured query for manual and reliable offset management?
  27. How to calculate Median in spark sqlContext for column of data type double
  28. Spark: disk I/O on stage boundaries explanation
  29. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment