ImportError: No module named numpy on spark workers

by

To use Spark in Yarn client mode, you’ll need to install any dependencies to the machines on which Yarn starts the executors. That’s the only surefire way to make this work.

Using Spark with Yarn cluster mode is a different story. You can distribute python dependencies with spark-submit. 

spark-submit --master yarn-cluster my_script.py --py-files my_dependency.zip

However, the situation with numpy is complicated by the same thing that makes it so fast: the fact that does the heavy lifting in C. Because of the way that it is installed, you won’t be able to distribute numpy in this fashion.

More Related Contents:

  1. Convert pyspark string to date format
  2. How to split Vector into columns – using PySpark
  3. How to use JDBC source to write and read data in (Py)Spark?
  4. Applying UDFs on GroupedData in PySpark (with functioning python example)
  5. Pyspark: Split multiple array columns into rows
  6. creating spark data structure from multiline record
  7. How to change a dataframe column from String type to Double type in PySpark?
  8. collect_list by preserving order based on another variable
  9. Retrieve top n in each group of a DataFrame in pyspark
  10. Pyspark: explode json in column to multiple columns
  11. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  12. How to use a Scala class inside Pyspark
  13. How to add third-party Java JAR files for use in PySpark
  14. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  15. Filter Pyspark dataframe column with None value
  16. PySpark logging from the executor
  17. Cast column containing multiple string date formats to DateTime in Spark
  18. How to explode multiple columns of a dataframe in pyspark
  19. Convert spark DataFrame column to python list
  20. PySpark: multiple conditions in when clause
  21. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  22. Load CSV file with PySpark
  23. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  24. How to pass a constant value to Python UDF?
  25. How to zip two array columns in Spark SQL
  26. Concatenate two PySpark dataframes
  27. Filtering DataFrame using the length of a column
  28. How to process RDDs using a Python class?
  29. What is the Spark DataFrame method `toPandas` actually doing?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment