What is the difference between Apache Spark SQLContext vs HiveContext?

by

Spark 2.0+

Spark 2.0 provides native window functions (SPARK-8641) and features some additional improvements in parsing and much better SQL 2003 compliance so it is significantly less dependent on Hive to achieve core funcionality and because of that HiveContext (SparkSession with Hive support) seems to be slightly less important.

Spark < 2.0

Obviously if you want to work with Hive you have to use HiveContext. Beyond that the biggest difference as for now (Spark 1.5) is a support for window functions and ability to access Hive UDFs.

Generally speaking window functions are a pretty cool feature and can be used to solve quite complex problems in a concise way without going back and forth between RDDs and DataFrames. Performance is still far from optimal especially without PARTITION BY clause but it is really nothing Spark specific.

Regarding Hive UDFs it is not a serious issue now, but before Spark 1.5 many SQL functions have been expressed using Hive UDFs and required HiveContext to work.

HiveContext also provides more robust SQL parser. See for example: py4j.protocol.Py4JJavaError when selecting nested column in dataframe using select statetment

Finally HiveContext is required to start Thrift server.

The biggest problem with HiveContext is that it comes with large dependencies.

More Related Contents:

  1. How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?
  2. How to control partition size in Spark SQL
  3. How to optimize partitioning when migrating data from JDBC source?
  4. How to connect to remote hive server from spark [duplicate]
  5. Use collect_list and collect_set in Spark SQL
  6. How to calculate Median in spark sqlContext for column of data type double
  7. Finding duplicates from large data set using Apache Spark
  8. Using a column value as a parameter to a spark DataFrame function
  9. Find maximum row per group in Spark DataFrame
  10. While writing to hdfs path getting error java.io.IOException: Failed to rename
  11. Multiple Aggregate operations on the same column of a spark dataframe
  12. Partitioning in spark while reading from RDBMS via JDBC
  13. How to import multiple csv files in a single load?
  14. How to access element of a VectorUDT column in a Spark DataFrame?
  15. How to save DataFrame directly to Hive?
  16. How to save/insert each DStream into a permanent table
  17. Spark load data and add filename as dataframe column
  18. What is the maximum size for a broadcast object in Spark?
  19. PySpark: how to resample frequencies
  20. How does createOrReplaceTempView work in Spark?
  21. PySpark – get row number for each row in a group
  22. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  23. Keep only duplicates from a DataFrame regarding some field
  24. pyspark: count distinct over a window
  25. Overwrite only some partitions in a partitioned spark Dataset
  26. How to get Kafka offsets for structured query for manual and reliable offset management?
  27. How to use groupBy to collect rows into a map?
  28. When to use Spark DataFrame/Dataset API and when to use plain RDD?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment