How to create an empty DataFrame with a specified schema?

by

Lets assume you want a data frame with the following schema:

root
 |-- k: string (nullable = true)
 |-- v: integer (nullable = false)

You simply define schema for a data frame and use empty RDD[Row]:

import org.apache.spark.sql.types.{
    StructType, StructField, StringType, IntegerType}
import org.apache.spark.sql.Row

val schema = StructType(
    StructField("k", StringType, true) ::
    StructField("v", IntegerType, false) :: Nil)

// Spark < 2.0
// sqlContext.createDataFrame(sc.emptyRDD[Row], schema) 
spark.createDataFrame(sc.emptyRDD[Row], schema)

PySpark equivalent is almost identical:

from pyspark.sql.types import StructType, StructField, IntegerType, StringType

schema = StructType([
    StructField("k", StringType(), True), StructField("v", IntegerType(), False)
])

# or df = sc.parallelize([]).toDF(schema)

# Spark < 2.0 
# sqlContext.createDataFrame([], schema)
df = spark.createDataFrame([], schema)

Using implicit encoders (Scala only) with Product types like Tuple:

import spark.implicits._

Seq.empty[(String, Int)].toDF("k", "v")

or case class:

case class KV(k: String, v: Int)

Seq.empty[KV].toDF

or

spark.emptyDataset[KV].toDF

More Related Contents:

  1. How to select the first row of each group?
  2. How to pivot Spark DataFrame?
  3. How to query JSON data column using Spark DataFrames?
  4. Difference between DataFrame, Dataset, and RDD in Spark
  5. How to define partitioning of DataFrame?
  6. How do I detect if a Spark DataFrame has a column
  7. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  8. How to split a dataframe into dataframes with same column values?
  9. Provide schema while reading csv file as a dataframe
  10. Dropping a nested column from Spark DataFrame
  11. Difference between DataFrame, Dataset, and RDD in Spark
  12. Filter Spark DataFrame based on another DataFrame that specifies denylist criteria
  13. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  14. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  15. How to zip two (or more) DataFrame in Spark
  16. Apache Spark how to append new column from list/array to Spark dataframe
  17. Spark Dataframes UPSERT to Postgres Table
  18. Create new Dataframe with empty/null field values
  19. Provide schema while reading csv file as a dataframe in Scala Spark
  20. Renaming column names of a DataFrame in Spark Scala
  21. Derive multiple columns from a single column in a Spark DataFrame
  22. Replace missing values with mean – Spark Dataframe
  23. What is going wrong with `unionAll` of Spark `DataFrame`?
  24. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  25. How to show full column content in a Spark Dataframe?
  26. spark dataframe drop duplicates and keep first
  27. How to get keys and values from MapType column in SparkSQL DataFrame
  28. Spark DataFrames when udf functions do not accept large enough input variables
  29. Filter spark DataFrame on string contains
  30. Difference between === null and isNull in Spark DataDrame

Leave a Comment