Concatenate two PySpark dataframes

by

Maybe you can try creating the unexisting columns and calling union (unionAll for Spark 1.6 or lower):

from pyspark.sql.functions import lit

cols = ['id', 'uniform', 'normal', 'normal_2']    

df_1_new = df_1.withColumn("normal_2", lit(None)).select(cols)
df_2_new = df_2.withColumn("normal", lit(None)).select(cols)

result = df_1_new.union(df_2_new)

# To remove the duplicates:

result = result.dropDuplicates()

More Related Contents:

  1. Convert pyspark string to date format
  2. How to split Vector into columns – using PySpark
  3. How to use JDBC source to write and read data in (Py)Spark?
  4. Load CSV file with Spark
  5. Applying UDFs on GroupedData in PySpark (with functioning python example)
  6. Pyspark: Split multiple array columns into rows
  7. How to change a dataframe column from String type to Double type in PySpark?
  8. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  9. Retrieve top n in each group of a DataFrame in pyspark
  10. Pyspark: explode json in column to multiple columns
  11. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  12. How to use a Scala class inside Pyspark
  13. Filter Pyspark dataframe column with None value
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. PySpark: multiple conditions in when clause
  17. Apache Spark Python Cosine Similarity over DataFrames
  18. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  19. Load CSV file with PySpark
  20. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  21. Cannot find col function in pyspark
  22. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  23. How to pass a constant value to Python UDF?
  24. How to zip two array columns in Spark SQL
  25. When to cache a DataFrame?
  26. Multiple condition filter on dataframe
  27. How to return a “Tuple type” in a UDF in PySpark?
  28. How to connect HBase and Spark using Python?
  29. How to pivot on multiple columns in Spark SQL?
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment