How to determine if object is a valid key-value pair in PySpark

by

Python is a dynamically typed language and PySpark doesn’t use any special type for key, value pairs. The only requirement for an object being considered a valid data for PairRDD operations is that it can be unpacked as follows:

k, v = kv

Typically you would use a two element tuple due to its semantics (immutable object of fixed size) and similarity to Scala Product classes. But this is just a convention and nothing stops you from something like this:

key_value.py

class KeyValue(object):
    def __init__(self, k, v):
        self.k = k
        self.v = v
    def __iter__(self):
       for x in [self.k, self.v]:
           yield x

from key_value import KeyValue

rdd = sc.parallelize(
    [KeyValue("foo", 1), KeyValue("foo", 2), KeyValue("bar", 0)]) 

rdd.reduceByKey(add).collect()
## [('bar', 0), ('foo', 3)]

and make an arbitrary class behave like a key-value. So once again if something can be correctly unpacked as a pair of objects then it is a valid key-value. Implementing __len__ and __getitem__ magic methods should work as well. Probably the most elegant way to handle this is to use namedtuples.

Also type(rdd.take(1)) returns a list of length n so its type will be always the same.

More Related Contents:

  1. How to find median and quantiles using Spark
  2. Spark Dataframe distinguish columns with duplicated name
  3. How to change dataframe column names in pyspark?
  4. I can’t seem to get –py-files on Spark to work
  5. Retrieve top n in each group of a DataFrame in pyspark
  6. Pyspark: explode json in column to multiple columns
  7. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  8. Spark RDD to DataFrame python
  9. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  10. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  11. PySpark logging from the executor
  12. Cast column containing multiple string date formats to DateTime in Spark
  13. How to explode multiple columns of a dataframe in pyspark
  14. How to extract an element from a array in pyspark
  15. PySpark: multiple conditions in when clause
  16. Create Spark DataFrame. Can not infer schema for type
  17. How to transform data with sliding window over time series data in Pyspark
  18. Create single row dataframe from list of list PySpark
  19. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  20. ImportError: No module named numpy on spark workers
  21. Build a hierarchy from a relational data-set using Pyspark
  22. Multiple condition filter on dataframe
  23. How to return a “Tuple type” in a UDF in PySpark?
  24. How to connect HBase and Spark using Python?
  25. How to process RDDs using a Python class?
  26. Pyspark 2.4.0, read avro from kafka with read stream – Python
  27. How to pivot on multiple columns in Spark SQL?
  28. Apache Spark — Assign the result of UDF to multiple dataframe columns
  29. What is the Spark DataFrame method `toPandas` actually doing?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment