What does setMaster `local[*]` mean in spark?

by

[*]

From the doc:

./bin/spark-shell --master local[2]

The --master option specifies the master URL for a distributed
cluster, or local to run locally with one thread, or local[N] to run
locally with N threads. You should start by using local for testing.

And from here:

local[*] Run Spark locally with as many worker threads as logical
cores on your machine.

More Related Contents:

  1. How to define and use a User-Defined Aggregate Function in Spark SQL?
  2. (Why) do we need to call cache or persist on a RDD
  3. Automatically and Elegantly flatten DataFrame in Spark SQL
  4. How do I skip a header from CSV files in Spark?
  5. Spark extracting values from a Row
  6. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  7. Provide schema while reading csv file as a dataframe
  8. How to flatmap a nested Dataframe in Spark
  9. How to save DataFrame directly to Hive?
  10. About how to add a new column to an existing DataFrame with random values in Scala
  11. How to use Column.isin with list?
  12. Modify collection inside a Spark RDD foreach
  13. Function returns an empty List in Spark
  14. How to compare two dataframe and print columns that are different in scala
  15. Operate on neighbor elements in RDD in Spark
  16. Customize SparkContext using sparkConf.set(..) when using spark-shell
  17. Renaming column names of a DataFrame in Spark Scala
  18. What is going wrong with `unionAll` of Spark `DataFrame`?
  19. Spark: Transpose DataFrame Without Aggregating
  20. Spark: long delay between jobs
  21. How to get keys and values from MapType column in SparkSQL DataFrame
  22. How to compute cumulative sum using Spark
  23. Addition of two RDD[mllib.linalg.Vector]’s
  24. Add Number of days column to Date Column in same dataframe for Spark Scala App
  25. How can I connect to a postgreSQL database into Apache Spark using scala?
  26. Spark column string replace when present in other column (row)
  27. How to vectorize DataFrame columns for ML algorithms?
  28. Spark DataFrames when udf functions do not accept large enough input variables
  29. Filter spark DataFrame on string contains
  30. Spark textFile vs wholeTextFiles

Leave a Comment