A list as a key for PySpark’s reduceByKey

by

Try this:

rdd.map(lambda (k, v): (tuple(k), v)).groupByKey()

Since Python lists are mutable it means that cannot be hashed (don’t provide __hash__ method):

>>> a_list = [1, 2, 3]
>>> a_list.__hash__ is None
True
>>> hash(a_list)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

Tuples from the other hand are immutable and provide __hash__ method implementation:

>>> a_tuple = (1, 2, 3)
>>> a_tuple.__hash__ is None
False
>>> hash(a_tuple)
2528502973977326415

hence can be used as a key. Similarly if you want to use unique values as a key you should use frozenset:

rdd.map(lambda (k, v): (frozenset(k), v)).groupByKey().collect()

instead of set.

# This will fail with TypeError: unhashable type: 'set'
rdd.map(lambda (k, v): (set(k), v)).groupByKey().collect()

More Related Contents:

  1. How to find median and quantiles using Spark
  2. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  3. Spark union of multiple RDDs
  4. Spark groupByKey alternative
  5. How to extract an element from a array in pyspark
  6. PySpark DataFrames – way to enumerate without converting to Pandas?
  7. Reduce a key-value pair into a key-list pair with Apache Spark
  8. Random numbers generation in PySpark
  9. Spark Dataframe distinguish columns with duplicated name
  10. How to change dataframe column names in pyspark?
  11. I can’t seem to get –py-files on Spark to work
  12. Retrieve top n in each group of a DataFrame in pyspark
  13. Pyspark: explode json in column to multiple columns
  14. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  15. Spark RDD to DataFrame python
  16. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  17. Why is Apache-Spark – Python so slow locally as compared to pandas?
  18. Cast column containing multiple string date formats to DateTime in Spark
  19. PySpark: multiple conditions in when clause
  20. Create Spark DataFrame. Can not infer schema for type
  21. How to transform data with sliding window over time series data in Pyspark
  22. Create single row dataframe from list of list PySpark
  23. Build a hierarchy from a relational data-set using Pyspark
  24. When to cache a DataFrame?
  25. environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
  26. How to return a “Tuple type” in a UDF in PySpark?
  27. PySpark Throwing error Method __getnewargs__([]) does not exist
  28. PySpark Evaluation
  29. Calculating the averages for each KEY in a Pairwise (K,V) RDD in Spark with Python
  30. What is the difference between spark-submit and pyspark?

Leave a Comment