How to Access Hive via Python?

by

I believe the easiest way is to use PyHive.

To install you’ll need these libraries:

pip install sasl
pip install thrift
pip install thrift-sasl
pip install PyHive

Please note that although you install the library as PyHive, you import the module as pyhive, all lower-case.

If you’re on Linux, you may need to install SASL separately before running the above. Install the package libsasl2-dev using apt-get or yum or whatever package manager for your distribution. For Windows there are some options on GNU.org, you can download a binary installer. On a Mac SASL should be available if you’ve installed xcode developer tools (xcode-select --install in Terminal)

After installation, you can connect to Hive like this:

from pyhive import hive
conn = hive.Connection(host="YOUR_HIVE_HOST", port=PORT, username="YOU")

Now that you have the hive connection, you have options how to use it. You can just straight-up query:

cursor = conn.cursor()
cursor.execute("SELECT cool_stuff FROM hive_table")
for result in cursor.fetchall():
  use_result(result)

…or to use the connection to make a Pandas dataframe:

import pandas as pd
df = pd.read_sql("SELECT cool_stuff FROM hive_table", conn)

More Related Contents:

  1. Tell me how to replace python data, How can I read the a.txt file and make it in the following format? [closed]
  2. How to turn off INFO logging in Spark?
  3. How can PySpark be called in debug mode?
  4. Python read file as stream from HDFS
  5. How can I include a python package with Hadoop streaming job?
  6. How do I run graphx with Python / pyspark?
  7. Python swapping and slicing method
  8. Cartesian product of x and y array points into single array of 2D points
  9. Why use pip over easy_install?
  10. What is the difference between pip and conda?
  11. Indexing one array by another in numpy
  12. datetime dtypes in pandas read_csv
  13. How to merge a transparent png image with another image using PIL
  14. catching stdout in realtime from subprocess
  15. Tensorflow – ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float)
  16. ImportError in importing from sklearn: cannot import name check_build
  17. H14 error in heroku – “no web processes running”
  18. How to access the previous/next element in a for loop?
  19. in selenium web driver how to choose the correct iframe
  20. How do I concatenate files in Python?
  21. Execute a Python script post install using distutils / setuptools
  22. Searching a sequence in a NumPy array
  23. Django: OperationalError No Such Table
  24. How to run multiple functions at the same time?
  25. Autocomplete for OpenCV-Python in Windows not working
  26. Run multiple programs sequentially in one Windows command prompt?
  27. How does a Python set([]) check if two objects are equal? What methods does an object need to define to customise this?
  28. How to disable password request for a Jupyter notebook session?
  29. Django vs other Python web frameworks?
  30. Pandas Secondary Axis

Leave a Comment