Spark Error:expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)

by

The source of the problem is that object returned from the UDF doesn’t conform to the declared type. np.unique not only returns numpy.ndarray but also converts numerics to the corresponding NumPy types which are not compatible with DataFrame API. You can try something like this:

udf(lambda x: list(set(x)), ArrayType(IntegerType()))

or this (to keep order)

udf(lambda xs: list(OrderedDict((x, None) for x in xs)), 
    ArrayType(IntegerType()))

instead.

If you really want np.unique you have to convert the output:

udf(lambda x: np.unique(x).tolist(), ArrayType(IntegerType()))

More Related Contents:

  1. Spark functions vs UDF performance?
  2. Applying UDFs on GroupedData in PySpark (with functioning python example)
  3. Passing a data frame column and external list to udf under withColumn
  4. Using UDF ignores condition in when
  5. How to pass a constant value to Python UDF?
  6. Access Array column in Spark
  7. Apache Spark — Assign the result of UDF to multiple dataframe columns
  8. Using a column value as a parameter to a spark DataFrame function
  9. How to split Vector into columns – using PySpark
  10. Split Spark Dataframe string column into multiple columns
  11. Avoid performance impact of a single partition mode in Spark window functions
  12. Retrieve top n in each group of a DataFrame in pyspark
  13. Pyspark: explode json in column to multiple columns
  14. How to split a list to multiple columns in Pyspark?
  15. pyspark dataframe filter or include based on list
  16. How to use a Scala class inside Pyspark
  17. Cast column containing multiple string date formats to DateTime in Spark
  18. How to explode multiple columns of a dataframe in pyspark
  19. PySpark: multiple conditions in when clause
  20. PySpark – get row number for each row in a group
  21. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  22. Load CSV file with PySpark
  23. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  24. Keep only duplicates from a DataFrame regarding some field
  25. How to zip two array columns in Spark SQL
  26. How can I access python variable in Spark SQL?
  27. Multiple condition filter on dataframe
  28. How to return a “Tuple type” in a UDF in PySpark?
  29. How to connect HBase and Spark using Python?
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment