Shipping Python modules in pyspark to other nodes

by

If you can package your module into a .egg or .zip file, you should be able to list it in pyFiles when constructing your SparkContext (or you can add it later through sc.addPyFile).

For Python libraries that use setuptools, you can run python setup.py bdist_egg to build an egg distribution.

Another option is to install the library cluster-wide, either by using pip/easy_install on each machine or by sharing a Python installation over a cluster-wide filesystem (like NFS).

More Related Contents:

  1. How to add a constant column in a Spark DataFrame?
  2. How do I add a new column to a Spark DataFrame (using PySpark)?
  3. How to perform union on two DataFrames with different amounts of columns in spark?
  4. Does spark predicate pushdown work with JDBC?
  5. Explode in PySpark
  6. Python worker failed to connect back
  7. Filtering a Pyspark DataFrame with SQL-like IN clause
  8. Spark union of multiple RDDs
  9. Dividing complex rows of dataframe to simple rows in Pyspark
  10. Spark DataFrame TimestampType – how to get Year, Month, Day values from field?
  11. Why is Apache-Spark – Python so slow locally as compared to pandas?
  12. Spark add new column to dataframe with value from previous row
  13. Is Spark’s KMeans unable to handle bigdata?
  14. How to create a custom Estimator in PySpark
  15. Pyspark: Replacing value in a column by searching a dictionary
  16. Pivot String column on Pyspark Dataframe
  17. PySpark DataFrames – way to enumerate without converting to Pandas?
  18. Reduce a key-value pair into a key-list pair with Apache Spark
  19. Using UDF ignores condition in when
  20. How do I set the driver’s python version in spark?
  21. Converting Pandas dataframe into Spark dataframe error
  22. PySpark in iPython notebook raises Py4JJavaError when using count() and first()
  23. Concatenating string by rows in pyspark
  24. PySpark: match the values of a DataFrame column against another DataFrame column
  25. What is the equivalent to scala.util.Try in pyspark?
  26. How do I log from my Python Spark script
  27. When to cache a DataFrame?
  28. PySpark Throwing error Method __getnewargs__([]) does not exist
  29. PySpark Evaluation
  30. What is the difference between spark-submit and pyspark?

Leave a Comment