What is the Spark DataFrame method `toPandas` actually doing?

by

Using spark to read in a CSV file to pandas is quite a roundabout method for achieving the end goal of reading a CSV file into memory.

It seems like you might be misunderstanding the use cases of the technologies in play here.

Spark is for distributed computing (though it can be used locally). It’s generally far too heavyweight to be used for simply reading in a CSV file.

In your example, the sc.textFile method will simply give you a spark RDD that is effectively a list of text lines. This likely isn’t what you want. No type inference will be performed, so if you want to sum a column of numbers in your CSV file, you won’t be able to because they are still strings as far as Spark is concerned.

Just use pandas.read_csv and read the whole CSV into memory. Pandas will automatically infer the type of each column. Spark doesn’t do this.

Now to answer your questions:

Does it store the Pandas object to local memory:

Yes. toPandas() will convert the Spark DataFrame into a Pandas DataFrame, which is of course in memory.

Does Pandas low-level computation handled all by Spark

No. Pandas runs its own computations, there’s no interplay between spark and pandas, there’s simply some API compatibility.

Does it exposed all pandas dataframe functionality?

No. For example, Series objects have an interpolate method which isn’t available in PySpark Column objects. There are many many methods and functions that are in the pandas API that are not in the PySpark API.

Can I convert it toPandas and just be done with it, without so much touching DataFrame API?

Absolutely. In fact, you probably shouldn’t even use Spark at all in this case. pandas.read_csv will likely handle your use case unless you’re working with a huge amount of data.

Try to solve your problem with simple, low-tech, easy-to-understand libraries, and only go to something more complicated as you need it. Many times, you won’t need the more complex technology.

More Related Contents:

  1. Why is Apache-Spark – Python so slow locally as compared to pandas?
  2. How to zip two array columns in Spark SQL
  3. Calling Java/Scala function from a task
  4. Efficient string matching in Apache Spark
  5. Load CSV file with Spark
  6. How to link PyCharm with PySpark?
  7. How to turn off INFO logging in Spark?
  8. Does spark predicate pushdown work with JDBC?
  9. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  10. importing pyspark in python shell
  11. collect_list by preserving order based on another variable
  12. Rename nested field in spark dataframe
  13. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  14. How to determine if object is a valid key-value pair in PySpark
  15. Updating a dataframe column in spark
  16. How to run multiple jobs in one Sparkcontext from separate threads in PySpark?
  17. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  18. Best way to get the max value in a Spark dataframe column
  19. How to map features from the output of a VectorAssembler back to the column names in Spark ML?
  20. Apache Spark Python Cosine Similarity over DataFrames
  21. Comparing columns in Pyspark
  22. GroupBy column and filter rows with maximum value in Pyspark
  23. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  24. Cannot find col function in pyspark
  25. Add Jar to standalone pyspark
  26. How to pass a constant value to Python UDF?
  27. Concatenate two PySpark dataframes
  28. How to process RDDs using a Python class?
  29. How to pivot on multiple columns in Spark SQL?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment