Use collect_list and collect_set in Spark SQL

by

Spark 2.0+:

SPARK-10605 introduced native collect_list and collect_set implementation. SparkSession with Hive support or HiveContext are no longer required.

Spark 2.0-SNAPSHOT (before 2016-05-03):

You have to enable Hive support for a given SparkSession:

In Scala:

val spark = SparkSession.builder
  .master("local")
  .appName("testing")
  .enableHiveSupport()  // <- enable Hive support.
  .getOrCreate()

In Python:

spark = (SparkSession.builder
    .enableHiveSupport()
    .getOrCreate())

Spark < 2.0:

To be able to use Hive UDFs (see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF) you have use Spark built with Hive support (this is already covered when you use pre-built binaries what seems to be the case here) and initialize SparkContext using HiveContext.

In Scala:

import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.SQLContext

val sqlContext: SQLContext = new HiveContext(sc)

In Python:

from pyspark.sql import HiveContext

sqlContext = HiveContext(sc)

More Related Contents:

  1. How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?
  2. How to control partition size in Spark SQL
  3. How to optimize partitioning when migrating data from JDBC source?
  4. How to connect to remote hive server from spark [duplicate]
  5. What is the difference between Apache Spark SQLContext vs HiveContext?
  6. How to calculate Median in spark sqlContext for column of data type double
  7. How to melt Spark DataFrame?
  8. Spark SQL: apply aggregate functions to a list of columns
  9. Unpivot in spark-sql/pyspark
  10. Spark SQL replacement for MySQL’s GROUP_CONCAT aggregate function
  11. DataFrame join optimization – Broadcast Hash Join
  12. Overwrite specific partitions in spark dataframe write method
  13. Convert date from String to Date format in Dataframes
  14. How to check if spark dataframe is empty?
  15. What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?
  16. Spark DataFrame Schema Nullable Fields
  17. Multiple Spark applications with HiveContext
  18. multiple conditions for filter in spark data frames
  19. How do I add an persistent column of row ids to Spark DataFrame?
  20. Rename more than one column using withColumnRenamed
  21. Convert null values to empty array in Spark DataFrame
  22. DataFrame partitionBy to a single Parquet file (per partition)
  23. Temp table caching with spark-sql
  24. Spark lists all leaf node even in partitioned data
  25. How to exclude multiple columns in Spark dataframe in Python
  26. How to get Kafka offsets for structured query for manual and reliable offset management?
  27. How to use groupBy to collect rows into a map?
  28. When to use Spark DataFrame/Dataset API and when to use plain RDD?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment