How do I get Python libraries in pyspark?

by

In the Spark context try using:

SparkContext.addPyFile("module.py")  # Also supports .zip

Quoting from the docs:

Add a .py or .zip dependency for all tasks to be executed on this
SparkContext in the future. The path passed can be either a local
file, a file in HDFS (or other Hadoop-supported filesystems), or an
HTTP, HTTPS or FTP URI.

More Related Contents:

  1. python 2.7 how to get bigger number between (a, b) [closed]
  2. Sub-process in Python execute two task? [closed]
  3. How do I break an input into individual characters in python? [duplicate]
  4. input() error – NameError: name ‘…’ is not defined
  5. TypeError: ‘int’ object is not callable
  6. Reading a huge .csv file
  7. What is the most efficient way of finding all the factors of a number in Python?
  8. HTTP requests and JSON parsing in Python
  9. How to print a string at a fixed width?
  10. How to map numeric data into categories / bins in Pandas dataframe
  11. Python parse CSV ignoring comma with double-quotes
  12. How do I pass extra arguments to a Python decorator?
  13. Accessing attributes on literals work on all types, but not `int`; why? [duplicate]
  14. Securely storing environment variables in GAE with app.yaml
  15. Split string using a newline delimiter with Python [duplicate]
  16. Python 2.7 : Write to file instantly
  17. Simplest way to plot 3d surface given 3d points
  18. Python write to a file returns empty file
  19. Python reading from a file and saving to utf-8
  20. RegEx Get string between two strings that has line breaks
  21. How do I check if raw input is integer in python 2.7?
  22. Is there a need to close files that have no reference to them?
  23. How to extract an element from a array in pyspark
  24. Write and read a list from file
  25. Flask broken pipe with requests
  26. Wait until task is completed on Remote Machine through Python [duplicate]
  27. Python json.loads changes the order of the object
  28. How can I understand a .pyc file content
  29. How can I read pdf in python? [duplicate]
  30. Defining a default argument as a global variable

Leave a Comment