What is the difference between spark-submit and pyspark?

by

  1. If you built a spark application, you need to use spark-submit to run the application

    • The code can be written either in python/scala

    • The mode can be either local/cluster

  2. If you just want to test/run few individual commands, you can use the shell provided by spark

    • pyspark (for spark in python)
    • spark-shell (for spark in scala)

More Related Contents:

  1. Calling Java/Scala function from a task
  2. Efficient string matching in Apache Spark
  3. Load CSV file with Spark
  4. How to link PyCharm with PySpark?
  5. How to turn off INFO logging in Spark?
  6. Does spark predicate pushdown work with JDBC?
  7. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  8. importing pyspark in python shell
  9. collect_list by preserving order based on another variable
  10. Rename nested field in spark dataframe
  11. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  12. How to determine if object is a valid key-value pair in PySpark
  13. Updating a dataframe column in spark
  14. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  15. Best way to get the max value in a Spark dataframe column
  16. How to map features from the output of a VectorAssembler back to the column names in Spark ML?
  17. Apache Spark Python Cosine Similarity over DataFrames
  18. Comparing columns in Pyspark
  19. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  20. Cannot find col function in pyspark
  21. Add Jar to standalone pyspark
  22. How to pass a constant value to Python UDF?
  23. How to zip two array columns in Spark SQL
  24. Concatenate two PySpark dataframes
  25. How do I convert an array (i.e. list) column to Vector
  26. Filtering DataFrame using the length of a column
  27. Add column sum as new column in PySpark dataframe
  28. Pyspark – converting json string to DataFrame
  29. How to drop all columns with null values in a PySpark DataFrame?
  30. Random numbers generation in PySpark

Leave a Comment