How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?

by

For Spark 1.x, you can set with :

System.setProperty("hive.metastore.uris", "thrift://METASTORE:9083");

final SparkConf conf = new SparkConf();
SparkContext sc = new SparkContext(conf);
HiveContext hiveContext = new HiveContext(sc);

Or 

final SparkConf conf = new SparkConf();
SparkContext sc = new SparkContext(conf);
HiveContext hiveContext = new HiveContext(sc);
hiveContext.setConf("hive.metastore.uris", "thrift://METASTORE:9083");

Update If your Hive is Kerberized :

Try setting these before creating the HiveContext :

System.setProperty("hive.metastore.sasl.enabled", "true");
System.setProperty("hive.security.authorization.enabled", "false");
System.setProperty("hive.metastore.kerberos.principal", hivePrincipal);
System.setProperty("hive.metastore.execute.setugi", "true");

More Related Contents:

  1. How to control partition size in Spark SQL
  2. How to optimize partitioning when migrating data from JDBC source?
  3. How to connect to remote hive server from spark [duplicate]
  4. Use collect_list and collect_set in Spark SQL
  5. What is the difference between Apache Spark SQLContext vs HiveContext?
  6. How to calculate Median in spark sqlContext for column of data type double
  7. Find maximum row per group in Spark DataFrame
  8. Partitioning in spark while reading from RDBMS via JDBC
  9. Spark SQL – load data with JDBC using SQL statement, not table name
  10. How to access element of a VectorUDT column in a Spark DataFrame?
  11. How to save DataFrame directly to Hive?
  12. Filtering a spark dataframe based on date
  13. How to save/insert each DStream into a permanent table
  14. Spark DataFrame Schema Nullable Fields
  15. Spark load data and add filename as dataframe column
  16. PySpark: how to resample frequencies
  17. Why does Spark think this is a cross / Cartesian join
  18. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  19. Spark SQL broadcast hash join
  20. What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?
  21. pyspark: count distinct over a window
  22. Spark lists all leaf node even in partitioned data
  23. Array Intersection in Spark SQL
  24. reduce result datasets into single dataset
  25. How to get Kafka offsets for structured query for manual and reliable offset management?
  26. Stratified sampling with pyspark
  27. How to use groupBy to collect rows into a map?
  28. When to use Spark DataFrame/Dataset API and when to use plain RDD?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment