How to loop through each row of dataFrame in pyspark

by

You simply cannot. DataFrames, same as other distributed data structures, are not iterable and can be accessed using only dedicated higher order function and / or SQL methods.

You can of course collect 

for row in df.rdd.collect():
    do_something(row)

or convert toLocalIterator 

for row in df.rdd.toLocalIterator():
    do_something(row)

and iterate locally as shown above, but it beats all purpose of using Spark.

More Related Contents:

  1. How to access element of a VectorUDT column in a Spark DataFrame?
  2. Why does Spark think this is a cross / Cartesian join
  3. How to exclude multiple columns in Spark dataframe in Python
  4. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  5. How to melt Spark DataFrame?
  6. Using a column value as a parameter to a spark DataFrame function
  7. Split Spark Dataframe string column into multiple columns
  8. DataFrame join optimization – Broadcast Hash Join
  9. Pyspark: Split multiple array columns into rows
  10. Avoid performance impact of a single partition mode in Spark window functions
  11. How to change a dataframe column from String type to Double type in PySpark?
  12. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  13. Retrieve top n in each group of a DataFrame in pyspark
  14. How to split a list to multiple columns in Pyspark?
  15. pyspark dataframe filter or include based on list
  16. Filter Pyspark dataframe column with None value
  17. How to explode multiple columns of a dataframe in pyspark
  18. What is the maximum size for a broadcast object in Spark?
  19. PySpark: multiple conditions in when clause
  20. Fill in null with previously known good value with pyspark
  21. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  22. How do I add an persistent column of row ids to Spark DataFrame?
  23. Rename more than one column using withColumnRenamed
  24. Convert null values to empty array in Spark DataFrame
  25. Keep only duplicates from a DataFrame regarding some field
  26. How can I access python variable in Spark SQL?
  27. Filtering DataFrame using the length of a column
  28. Stratified sampling with pyspark
  29. Why is predicate pushdown not used in typed Dataset API (vs untyped DataFrame API)?
  30. How to turn off scientific notation in pyspark?

Leave a Comment