I can’t seem to get –py-files on Spark to work

by

First off, I’ll assume that your dependencies are listed in requirements.txt. To package and zip the dependencies, run the following at the command line:

pip install -t dependencies -r requirements.txt
cd dependencies
zip -r ../dependencies.zip .

Above, the cd dependencies command is crucial to ensure that the modules are the in the top level of the zip file. Thanks to Dan Corin’s post for heads up.

Next, submit the job via:

spark-submit --py-files dependencies.zip spark_job.py

The --py-files directive sends the zip file to the Spark workers but does not add it to the PYTHONPATH (source of confusion for me). To add the dependencies to the PYTHONPATH to fix the ImportError, add the following line to the Spark job, spark_job.py:

sc.addPyFile("dependencies.zip")

A caveat from this Cloudera post:

An assumption that anyone doing distributed computing with commodity
hardware must assume is that the underlying hardware is potentially
heterogeneous. A Python egg built on a client machine will be specific
to the client’s CPU architecture because of the required C
compilation. Distributing an egg for a complex, compiled package like
NumPy, SciPy, or pandas is a brittle solution that is likely to fail
on most clusters, at least eventually.

Although the solution above does not build an egg, the same guideline applies.

More Related Contents:

  1. How to split Vector into columns – using PySpark
  2. How to find median and quantiles using Spark
  3. Spark Dataframe distinguish columns with duplicated name
  4. creating spark data structure from multiline record
  5. How to change dataframe column names in pyspark?
  6. Retrieve top n in each group of a DataFrame in pyspark
  7. Pyspark: explode json in column to multiple columns
  8. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  9. How to use a Scala class inside Pyspark
  10. How to add third-party Java JAR files for use in PySpark
  11. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  12. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  13. Filter Pyspark dataframe column with None value
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. How to extract an element from a array in pyspark
  17. PySpark: multiple conditions in when clause
  18. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  19. Create Spark DataFrame. Can not infer schema for type
  20. Load CSV file with PySpark
  21. How to transform data with sliding window over time series data in Pyspark
  22. Create single row dataframe from list of list PySpark
  23. Build a hierarchy from a relational data-set using Pyspark
  24. Concatenate two PySpark dataframes
  25. How do I convert an array (i.e. list) column to Vector
  26. Filtering DataFrame using the length of a column
  27. How to process RDDs using a Python class?
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. What is the Spark DataFrame method `toPandas` actually doing?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment