Spark unionAll multiple dataframes

by

For pyspark you can do the following:

from functools import reduce
from pyspark.sql import DataFrame

dfs = [df1,df2,df3]
df = reduce(DataFrame.unionAll, dfs)

It’s also worth nothing that the order of the columns in the dataframes should be the same for this to work. This can silently give unexpected results if you don’t have the correct column orders!!

If you are using pyspark 2.3 or greater, you can use unionByName so you don’t have to reorder the columns.

More Related Contents:

  1. Spark 2.0 Dataset vs DataFrame
  2. Spark – load CSV file as DataFrame?
  3. How to define partitioning of DataFrame?
  4. Automatically and Elegantly flatten DataFrame in Spark SQL
  5. Spark extracting values from a Row
  6. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  7. SparkSQL: How to deal with null values in user defined function?
  8. How to aggregate values into collection after groupBy?
  9. How to read records in JSON format from Kafka using Structured Streaming?
  10. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  11. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  12. Processing multiple files as independent RDD’s in parallel
  13. MatchError while accessing vector column in Spark 2.0
  14. How to save DataFrame directly to Hive?
  15. Spark / Scala: forward fill with last observation
  16. About how to add a new column to an existing DataFrame with random values in Scala
  17. How to use Column.isin with list?
  18. How to compare two dataframe and print columns that are different in scala
  19. Perform a typed join in Scala with Spark Datasets
  20. Apache Spark: Get number of records per partition
  21. Renaming column names of a DataFrame in Spark Scala
  22. DataFrame-ified zipWithIndex
  23. Derive multiple columns from a single column in a Spark DataFrame
  24. How to save a spark DataFrame as csv on disk?
  25. What is going wrong with `unionAll` of Spark `DataFrame`?
  26. How to sort by column in descending order in Spark SQL?
  27. How to get keys and values from MapType column in SparkSQL DataFrame
  28. Why is join not possible after show operator?
  29. How to create DataFrame from Scala’s List of Iterables?
  30. How to create a Dataset of Maps?

Leave a Comment