Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error

by

This is a great example of why you shouldn’t use import *.

The line

from pyspark.sql.functions import *

will bring in all the functions in the pyspark.sql.functions module into your namespace, include some that will shadow your builtins.

The specific issue is in the count_elements function on the line:

n = sum(1 for _ in iterator)
#   ^^^ - this is now pyspark.sql.functions.sum

You intended to call __builtin__.sum, but the import * shadowed the builtin.

Instead, do one of the following:

import pyspark.sql.functions as f

Or 

from pyspark.sql.functions import sum as sum_

More Related Contents:

  1. Spark Equivalent of IF Then ELSE
  2. How to add a constant column in a Spark DataFrame?
  3. Spark Dataframe distinguish columns with duplicated name
  4. How do I add a new column to a Spark DataFrame (using PySpark)?
  5. How to perform union on two DataFrames with different amounts of columns in spark?
  6. Does spark predicate pushdown work with JDBC?
  7. How to change dataframe column names in pyspark?
  8. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  9. Explode in PySpark
  10. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  11. Passing a data frame column and external list to udf under withColumn
  12. Updating a dataframe column in spark
  13. Dividing complex rows of dataframe to simple rows in Pyspark
  14. Why is Apache-Spark – Python so slow locally as compared to pandas?
  15. Spark add new column to dataframe with value from previous row
  16. PySpark: multiple conditions in when clause
  17. PySpark create new column with mapping from a dict
  18. Pyspark: Replacing value in a column by searching a dictionary
  19. Create Spark DataFrame. Can not infer schema for type
  20. Pivot String column on Pyspark Dataframe
  21. Using UDF ignores condition in when
  22. Create single row dataframe from list of list PySpark
  23. Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
  24. How do I convert an array (i.e. list) column to Vector
  25. Add column sum as new column in PySpark dataframe
  26. When to cache a DataFrame?
  27. How to add suffix and prefix to all columns in python/pyspark dataframe
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. Pyspark changing type of column from date to string
  30. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe

Leave a Comment