Using windowing functions in Spark

by

I have already answered a similar question before. The error message says all. With spark < version 2.x, you’ll need a HiveContext in your application jar, no other way around.

You can read further about the difference between SQLContextand HiveContext here.

SparkSQL has a SQLContext and a HiveContext. HiveContext is a super set of the SQLContext. The Spark community suggest using the HiveContext. You can see that when you run spark-shell, which is your interactive driver application, it automatically creates a SparkContext defined as sc and a HiveContext defined as sqlContext. The HiveContext allows you to execute SQL queries as well as Hive commands.

You can try to check that inside of your spark-shell :

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_74)

scala> sqlContext.isInstanceOf[org.apache.spark.sql.hive.HiveContext]
res0: Boolean = true

scala> sqlContext.isInstanceOf[org.apache.spark.sql.SQLContext]
res1: Boolean = true

scala> sqlContext.getClass.getName
res2: String = org.apache.spark.sql.hive.HiveContext

By inheritance, HiveContext is actually an SQLContext, but it’s not true the other way around. You can check the source code if you are more intersted in knowing how does HiveContext inherits from SQLContext.

Since spark 2.0, you’ll just need to create a SparkSession (as the single entry point) which removes the HiveContext/SQLContext confusion issue.

More Related Contents:

  1. Avoid performance impact of a single partition mode in Spark window functions
  2. pyspark: count distinct over a window
  3. Finding duplicates from large data set using Apache Spark
  4. Using a column value as a parameter to a spark DataFrame function
  5. Find maximum row per group in Spark DataFrame
  6. Split Spark Dataframe string column into multiple columns
  7. While writing to hdfs path getting error java.io.IOException: Failed to rename
  8. Multiple Aggregate operations on the same column of a spark dataframe
  9. Partitioning in spark while reading from RDBMS via JDBC
  10. Spark SQL window function with complex condition
  11. How to control partition size in Spark SQL
  12. How to import multiple csv files in a single load?
  13. How to access element of a VectorUDT column in a Spark DataFrame?
  14. How to split a list to multiple columns in Pyspark?
  15. How to save/insert each DStream into a permanent table
  16. Spark load data and add filename as dataframe column
  17. What is the maximum size for a broadcast object in Spark?
  18. PySpark: how to resample frequencies
  19. Generate a Spark StructType / Schema from a case class
  20. How does createOrReplaceTempView work in Spark?
  21. PySpark – get row number for each row in a group
  22. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  23. Keep only duplicates from a DataFrame regarding some field
  24. Array Intersection in Spark SQL
  25. PySpark: How to fillna values in dataframe for specific columns?
  26. How to get Kafka offsets for structured query for manual and reliable offset management?
  27. How to calculate Median in spark sqlContext for column of data type double
  28. Spark: disk I/O on stage boundaries explanation
  29. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment