Why can’t PySpark find py4j.java_gateway?

by

In my environment (using docker and the image sequenceiq/spark:1.1.0-ubuntu), I ran in to this. If you look at the pyspark shell script, you’ll see that you need a few things added to your PYTHONPATH:

export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH

That worked in ipython for me.

Update: as noted in the comments, the name of the py4j zip file changes with each Spark release, so look around for the right name.

More Related Contents:

  1. IPython Notebook locale error [duplicate]
  2. Using both Python 2.x and Python 3.x in IPython Notebook
  3. How to display pandas DataFrame of floats using a format string for columns?
  4. How can PySpark be called in debug mode?
  5. What is the right way to debug in iPython notebook?
  6. How to add third-party Java JAR files for use in PySpark
  7. R and Python in one Jupyter notebook
  8. Jupyter | The kernel appears to have died. It will restart automatically |
  9. Auto-Loading a module on IPython startup
  10. Creating multiple python dictionaries from a file
  11. Why doesn’t Python run my bash code?
  12. What is the result of % in Python?
  13. Purpose of “%matplotlib inline”
  14. List on python appending always the same value [duplicate]
  15. sklearn error ValueError: Input contains NaN, infinity or a value too large for dtype(‘float64’)
  16. ImportError: numpy.core.multiarray failed to import
  17. Display realtime output of a subprocess in a tkinter widget
  18. How to install xgboost package in python (windows platform)?
  19. Matplotlib issue on OS X (“ImportError: cannot import name _thread”)
  20. How to use __setattr__ correctly, avoiding infinite recursion
  21. How to run multiple jobs in one Sparkcontext from separate threads in PySpark?
  22. How to get selectors with dynamic part inside using Selenium with Python?
  23. ValueError: malformed string when using ast.literal_eval
  24. Python string literal concatenation
  25. Rounding to two decimal places in Python 2.7?
  26. Python: how to capture image from webcam on click using OpenCV
  27. Parse a Pandas column to Datetime when importing table from SQL database and filtering rows by date
  28. Display an image with Python
  29. How to process RDDs using a Python class?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment