Spark sql queries vs dataframe functions

by

There is no performance difference whatsoever. Both methods use exactly the same execution engine and internal data structures. At the end of the day, all boils down to personal preferences.

  • Arguably DataFrame queries are much easier to construct programmatically and provide a minimal type safety.

  • Plain SQL queries can be significantly more concise and easier to understand. They are also portable and can be used without any modifications with every supported language. With HiveContext, these can also be used to expose some functionalities which can be inaccessible in other ways (for example UDF without Spark wrappers).

More Related Contents:

  1. How to select the first row of each group?
  2. Querying Spark SQL DataFrame with complex types
  3. Concatenate columns in Apache Spark DataFrame
  4. How to get other columns when using Spark DataFrame groupby?
  5. Difference between === null and isNull in Spark DataDrame
  6. DataFrame / Dataset groupBy behaviour/optimization
  7. Spark Window Functions – rangeBetween dates
  8. Spark SQL window function with complex condition
  9. Does SparkSQL support subquery?
  10. Including null values in an Apache Spark Join
  11. Explode (transpose?) multiple columns in Spark SQL table
  12. get TopN of all groups after group by using Spark DataFrame
  13. Why using a UDF in a SQL query leads to cartesian product?
  14. How to aggregate over rolling time window with groups in Spark
  15. What is the difference between cube, rollup and groupBy operators?
  16. MySQL INSERT INTO table VALUES.. vs INSERT INTO table SET
  17. DataFrame join optimization – Broadcast Hash Join
  18. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  19. How to split a dataframe into dataframes with same column values?
  20. How to create an empty DataFrame with a specified schema?
  21. What’s the best to check if item exist or not: Select Count(ID)OR Exist(…)?
  22. Apache Spark how to append new column from list/array to Spark dataframe
  23. SQL count(*) performance
  24. Why is a UDF so much slower than a subquery?
  25. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  26. How do I add an persistent column of row ids to Spark DataFrame?
  27. Improving performance of cluster index GUID primary key
  28. Convert null values to empty array in Spark DataFrame
  29. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  30. How do you interpret a query’s explain plan?

Leave a Comment