How to extract an element from a array in pyspark

by

Create sample data:

from pyspark.sql import Row
x = [Row(col1="xx", col2="yy", col3="zz", col4=[123,234])]
rdd = sc.parallelize([Row(col1="xx", col2="yy", col3="zz", col4=[123,234])])
df = spark.createDataFrame(rdd)
df.show()
#+----+----+----+----------+
#|col1|col2|col3|      col4|
#+----+----+----+----------+
#|  xx|  yy|  zz|[123, 234]|
#+----+----+----+----------+

Use getItem to extract element from the array column as this, in your actual case replace col4 with collect_set(TIMESTAMP):

df = df.withColumn("col5", df["col4"].getItem(1)).withColumn("col4", df["col4"].getItem(0))
df.show()
#+----+----+----+----+----+
#|col1|col2|col3|col4|col5|
#+----+----+----+----+----+
#|  xx|  yy|  zz| 123| 234|
#+----+----+----+----+----+

More Related Contents:

  1. How to find median and quantiles using Spark
  2. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  3. Spark union of multiple RDDs
  4. Spark groupByKey alternative
  5. A list as a key for PySpark’s reduceByKey
  6. PySpark DataFrames – way to enumerate without converting to Pandas?
  7. Reduce a key-value pair into a key-list pair with Apache Spark
  8. Random numbers generation in PySpark
  9. Spark Dataframe distinguish columns with duplicated name
  10. How to change dataframe column names in pyspark?
  11. I can’t seem to get –py-files on Spark to work
  12. Pyspark: explode json in column to multiple columns
  13. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  14. Spark RDD to DataFrame python
  15. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  16. Why is Apache-Spark – Python so slow locally as compared to pandas?
  17. Cast column containing multiple string date formats to DateTime in Spark
  18. PySpark: multiple conditions in when clause
  19. Create Spark DataFrame. Can not infer schema for type
  20. Load CSV file with PySpark
  21. How to transform data with sliding window over time series data in Pyspark
  22. Create single row dataframe from list of list PySpark
  23. Build a hierarchy from a relational data-set using Pyspark
  24. When to cache a DataFrame?
  25. environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
  26. Multiple condition filter on dataframe
  27. PySpark Throwing error Method __getnewargs__([]) does not exist
  28. PySpark Evaluation
  29. Calculating the averages for each KEY in a Pairwise (K,V) RDD in Spark with Python
  30. What is the difference between spark-submit and pyspark?

Leave a Comment