Concatenating string by rows in pyspark

by

The simplest way I can think of is to use collect_list

import pyspark.sql.functions as f
df.groupby("col1").agg(f.concat_ws(", ", f.collect_list(df.col2)))

More Related Contents:

  1. Calling Java/Scala function from a task
  2. Efficient string matching in Apache Spark
  3. Load CSV file with Spark
  4. How to link PyCharm with PySpark?
  5. How to turn off INFO logging in Spark?
  6. Does spark predicate pushdown work with JDBC?
  7. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  8. importing pyspark in python shell
  9. collect_list by preserving order based on another variable
  10. Rename nested field in spark dataframe
  11. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  12. How to determine if object is a valid key-value pair in PySpark
  13. Updating a dataframe column in spark
  14. How to run multiple jobs in one Sparkcontext from separate threads in PySpark?
  15. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  16. Best way to get the max value in a Spark dataframe column
  17. How to map features from the output of a VectorAssembler back to the column names in Spark ML?
  18. Pyspark replace strings in Spark dataframe column
  19. Apache Spark Python Cosine Similarity over DataFrames
  20. Comparing columns in Pyspark
  21. GroupBy column and filter rows with maximum value in Pyspark
  22. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  23. Cannot find col function in pyspark
  24. Add Jar to standalone pyspark
  25. Spark mllib predicting weird number or NaN
  26. Concatenate two PySpark dataframes
  27. Filtering DataFrame using the length of a column
  28. How to process RDDs using a Python class?
  29. What is the Spark DataFrame method `toPandas` actually doing?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment