How do I set the driver’s python version in spark?

by

Setting both PYSPARK_PYTHON=python3 and PYSPARK_DRIVER_PYTHON=python3 works for me.

I did this using export in my .bashrc. In the end, these are the variables I create:

export SPARK_HOME="$HOME/Downloads/spark-1.4.0-bin-hadoop2.4"
export IPYTHON=1
export PYSPARK_PYTHON=/usr/bin/python3
export PYSPARK_DRIVER_PYTHON=ipython3
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"

I also followed this tutorial to make it work from within Ipython3 notebook:
http://ramhiser.com/2015/02/01/configuring-ipython-notebook-support-for-pyspark/

More Related Contents:

  1. Convert pyspark string to date format
  2. How to split Vector into columns – using PySpark
  3. How to use JDBC source to write and read data in (Py)Spark?
  4. Applying UDFs on GroupedData in PySpark (with functioning python example)
  5. Pyspark: Split multiple array columns into rows
  6. creating spark data structure from multiline record
  7. How to change a dataframe column from String type to Double type in PySpark?
  8. Retrieve top n in each group of a DataFrame in pyspark
  9. Pyspark: explode json in column to multiple columns
  10. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  11. How to use a Scala class inside Pyspark
  12. How to add third-party Java JAR files for use in PySpark
  13. Filter Pyspark dataframe column with None value
  14. PySpark logging from the executor
  15. Cast column containing multiple string date formats to DateTime in Spark
  16. How to explode multiple columns of a dataframe in pyspark
  17. Convert spark DataFrame column to python list
  18. PySpark: multiple conditions in when clause
  19. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  20. Load CSV file with PySpark
  21. Cannot find col function in pyspark
  22. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  23. ImportError: No module named numpy on spark workers
  24. How do I convert an array (i.e. list) column to Vector
  25. Add column sum as new column in PySpark dataframe
  26. How to add suffix and prefix to all columns in python/pyspark dataframe
  27. Spark DAG differs with ‘withColumn’ vs ‘select’
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. Pyspark changing type of column from date to string
  30. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe

Leave a Comment