Multiple Spark applications with HiveContext

by

By default Hive(Context) is using embedded Derby as a metastore. It is intended mostly for testing and supports only one active user. If you want to support multiple running applications you should configure a standalone metastore. At this moment Hive supports PostgreSQL, MySQL, Oracle and MySQL. Details of configuration depend on a backend and option (local / remote) but generally speaking you’ll need:

Cloudera provides a comprehensive guide you may find useful: Configuring the Hive Metastore.

Theoretically it should be also possible to create separate Derby metastores with a proper configuration (see Hive Admin Manual – Local/Embedded Metastore Database) or to use Derby in Server Mode.

For development you can start applications in different working directories. This will create separate metastore_db for each application and avoid the issue of multiple active users. Providing separate Hive configuration should work as well but is less useful in development:

When not configured by the hive-site.xml, the context automatically creates metastore_db in the current directory

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Using a column value as a parameter to a spark DataFrame function
  3. Find maximum row per group in Spark DataFrame
  4. How do I split an RDD into two or more RDDs?
  5. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  6. Split Spark Dataframe string column into multiple columns
  7. How to control partition size in Spark SQL
  8. How to access element of a VectorUDT column in a Spark DataFrame?
  9. Python Spark Cumulative Sum by Group Using DataFrame
  10. How to split a list to multiple columns in Pyspark?
  11. Link Spark with iPython Notebook
  12. How to save/insert each DStream into a permanent table
  13. Spark load data and add filename as dataframe column
  14. PySpark: how to resample frequencies
  15. Efficient pyspark join
  16. Fill in null with previously known good value with pyspark
  17. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  18. PySpark – get row number for each row in a group
  19. Save ML model for future usage
  20. Apache spark dealing with case statements
  21. reading json file in pyspark
  22. pyspark: count distinct over a window
  23. Overwrite only some partitions in a partitioned spark Dataset
  24. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  25. Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:
  26. PySpark: How to fillna values in dataframe for specific columns?
  27. Spark + s3 – error – java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
  28. How to calculate Median in spark sqlContext for column of data type double
  29. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment