environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON

by

This may happen also if you’re working within an environment. In this case, it may be harder to retrieve the correct path to the python executable (and anyway I think it’s not a good idea to hardcode the path if you want to share it with others).

If you run the following lines at the beginning of your script/notebook (at least before you create the SparkSession/SparkContext) the problem is solved:

import os
import sys

os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable

Package os allows you to set global variables; package sys gives the string with the absolute path of the executable binary for the Python interpreter.

More Related Contents:

  1. Convert pyspark string to date format
  2. Calling Java/Scala function from a task
  3. How to use JDBC source to write and read data in (Py)Spark?
  4. How to link PyCharm with PySpark?
  5. Applying UDFs on GroupedData in PySpark (with functioning python example)
  6. Pyspark: Split multiple array columns into rows
  7. How to change a dataframe column from String type to Double type in PySpark?
  8. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  9. importing pyspark in python shell
  10. collect_list by preserving order based on another variable
  11. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  12. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  13. How to determine if object is a valid key-value pair in PySpark
  14. How to use a Scala class inside Pyspark
  15. How to add third-party Java JAR files for use in PySpark
  16. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  17. Filter Pyspark dataframe column with None value
  18. PySpark logging from the executor
  19. Convert spark DataFrame column to python list
  20. Comparing columns in Pyspark
  21. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  22. Load CSV file with PySpark
  23. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  24. Cannot find col function in pyspark
  25. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  26. ImportError: No module named numpy on spark workers
  27. How to process RDDs using a Python class?
  28. How to pivot on multiple columns in Spark SQL?
  29. What is the Spark DataFrame method `toPandas` actually doing?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment