When to use Spark DataFrame/Dataset API and when to use plain RDD?

by

From this Databricks’ blog article A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets

When to use RDDs?

Consider these scenarios or common use cases for
using RDDs when:

  • you want low-level transformation and actions and control on your
    dataset;
  • your data is unstructured, such as media streams or streams
    of text;
  • you want to manipulate your data with functional programming
    constructs than domain specific expressions;
  • you don’t care about
    imposing a schema, such as columnar format, while processing or
    accessing data attributes by name or column;
  • and you can forgo some
    optimization and performance benefits available with DataFrames and
    Datasets for structured and semi-structured data.

In High Performance Spark‘s Chapter 3. DataFrames, Datasets, and Spark SQL, you can see some performance you can get with the Dataframe/Dataset API compared to RDD

enter image description here

And in the Databricks’ article mentioned you can also find that Dataframe optimizes space usage compared to RDD

enter image description here

More Related Contents:

  1. Array Intersection in Spark SQL
  2. Why is predicate pushdown not used in typed Dataset API (vs untyped DataFrame API)?
  3. How to melt Spark DataFrame?
  4. Spark SQL: apply aggregate functions to a list of columns
  5. How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?
  6. Unpivot in spark-sql/pyspark
  7. Spark 2.0 Dataset vs DataFrame
  8. DataFrame / Dataset groupBy behaviour/optimization
  9. Spark SQL replacement for MySQL’s GROUP_CONCAT aggregate function
  10. Encoder error while trying to map dataframe row to updated row
  11. Spark SQL – load data with JDBC using SQL statement, not table name
  12. Spark Dataframe validating column names for parquet writes
  13. How to check if spark dataframe is empty?
  14. How to connect to remote hive server from spark [duplicate]
  15. Filtering a spark dataframe based on date
  16. Pyspark : forward fill with last observation for a DataFrame
  17. Spark DataFrame Schema Nullable Fields
  18. multiple conditions for filter in spark data frames
  19. Why does Spark think this is a cross / Cartesian join
  20. How do I add an persistent column of row ids to Spark DataFrame?
  21. Rename more than one column using withColumnRenamed
  22. Convert null values to empty array in Spark DataFrame
  23. DataFrame partitionBy to a single Parquet file (per partition)
  24. What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?
  25. Temp table caching with spark-sql
  26. PySpark: How to fillna values in dataframe for specific columns?
  27. Why spark application fail with “executor.CoarseGrainedExecutorBackend: Driver Disassociated”?
  28. What does “Correlated scalar subqueries must be Aggregated” mean?
  29. How to calculate Median in spark sqlContext for column of data type double
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment