Convert spark DataFrame column to python list

by

See, why this way that you are doing is not working. First, you are trying to get integer from a Row Type, the output of your collect is like this:

>>> mvv_list = mvv_count_df.select('mvv').collect()
>>> mvv_list[0]
Out: Row(mvv=1)

If you take something like this:

>>> firstvalue = mvv_list[0].mvv
Out: 1

You will get the mvv value. If you want all the information of the array you can take something like this:

>>> mvv_array = [int(row.mvv) for row in mvv_list.collect()]
>>> mvv_array
Out: [1,2,3,4]

But if you try the same for the other column, you get:

>>> mvv_count = [int(row.count) for row in mvv_list.collect()]
Out: TypeError: int() argument must be a string or a number, not 'builtin_function_or_method'

This happens because count is a built-in method. And the column has the same name as count. A workaround to do this is change the column name of count to _count:

>>> mvv_list = mvv_list.selectExpr("mvv as mvv", "count as _count")
>>> mvv_count = [int(row._count) for row in mvv_list.collect()]

But this workaround is not needed, as you can access the column using the dictionary syntax:

>>> mvv_array = [int(row['mvv']) for row in mvv_list.collect()]
>>> mvv_count = [int(row['count']) for row in mvv_list.collect()]

And it will finally work!

More Related Contents:

  1. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  2. Spark RDD to DataFrame python
  3. Convert pyspark string to date format
  4. How to split Vector into columns – using PySpark
  5. How to use JDBC source to write and read data in (Py)Spark?
  6. Applying UDFs on GroupedData in PySpark (with functioning python example)
  7. Pyspark: Split multiple array columns into rows
  8. creating spark data structure from multiline record
  9. How to change a dataframe column from String type to Double type in PySpark?
  10. Retrieve top n in each group of a DataFrame in pyspark
  11. Pyspark: explode json in column to multiple columns
  12. How to use a Scala class inside Pyspark
  13. How to add third-party Java JAR files for use in PySpark
  14. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  15. Filter Pyspark dataframe column with None value
  16. PySpark logging from the executor
  17. Cast column containing multiple string date formats to DateTime in Spark
  18. How to explode multiple columns of a dataframe in pyspark
  19. Comparing columns in Pyspark
  20. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  21. Load CSV file with PySpark
  22. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  23. Cannot find col function in pyspark
  24. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  25. ImportError: No module named numpy on spark workers
  26. How to pass a constant value to Python UDF?
  27. How to zip two array columns in Spark SQL
  28. When to cache a DataFrame?
  29. PySpark Throwing error Method __getnewargs__([]) does not exist
  30. PySpark Evaluation

Leave a Comment