How can I include a python package with Hadoop streaming job?

by

Just came across this gem of a solution: http://blog.cloudera.com/blog/2008/11/sending-files-to-remote-task-nodes-with-hadoop-mapreduce/

first create zip w/ the libraries desired

zip -r nltkandyaml.zip nltk yaml
mv ntlkandyaml.zip /path/to/where/your/mapper/will/be/nltkandyaml.mod

next, include via Hadoop stream “-file” argument:

hadoop -file nltkandyaml.zip

finally, load the libaries via python:

import zipimport
importer = zipimport.zipimporter('nltkandyaml.mod')
yaml = importer.load_module('yaml')
nltk = importer.load_module('nltk')

Additionally, this page summarizes how to include a corpus: http://www.xcombinator.com/2009/11/18/how-to-use-cascading-with-hadoop-streaming/

download and unzip the wordnet corpus

cd wordnet
zip -r ../wordnet-flat.zip *

in python:

wn = WordNetCorpusReader(nltk.data.find('lib/wordnet-flat.zip'))

More Related Contents:

  1. How to turn off INFO logging in Spark?
  2. How can PySpark be called in debug mode?
  3. Python read file as stream from HDFS
  4. How do I run graphx with Python / pyspark?
  5. How to Access Hive via Python?
  6. Clustered Stacked Bar in Python Pandas [closed]
  7. What to do with “Unexpected indent” in python?
  8. Replacing instances of a character in a string
  9. Saving a Numpy array as an image
  10. Convert pandas timezone-aware DateTimeIndex to naive timestamp, but in certain timezone
  11. How to install R packages that are not available in “R-essentials”?
  12. How to remove gaps between subplots in matplotlib
  13. Python RuntimeWarning: overflow encountered in long scalars
  14. Make function definition in a python file order independent [duplicate]
  15. Selecting Random Windows from Multidimensional Numpy Array Rows
  16. Python, Error while installing matplotlib
  17. Passing variables between methods in Python?
  18. How to have password echoed as asterisks [duplicate]
  19. What causes this error (AttributeError: ‘Mul’ object has no attribute ‘cos’) in Python?
  20. can you write a str.replace() using dictionary values in Python?
  21. How can I make sense of the `else` clause of Python loops?
  22. Reading multiple JSON records into a Pandas dataframe
  23. Copy directory contents into a directory with python [duplicate]
  24. How can I check if a string has a numeric value in it in Python? [duplicate]
  25. Can I dynamically convert an instance of one class to another?
  26. Remove non-ASCII characters from a string using python / django
  27. Why does a python descriptor __get__ method accept the owner class as an arg?
  28. subprocess.Popen in different console
  29. Delete multiple files matching a pattern
  30. python dictionary is thread safe?

Leave a Comment