How to pass -D parameter or environment variable to Spark job?

by

Change spark-submit command line adding three options:

  • --files <location_to_your_app.conf>
  • --conf 'spark.executor.extraJavaOptions=-Dconfig.resource=app'
  • --conf 'spark.driver.extraJavaOptions=-Dconfig.resource=app'

More Related Contents:

  1. Write single CSV file using spark-csv
  2. How to convert rdd object to dataframe in spark
  3. How to define and use a User-Defined Aggregate Function in Spark SQL?
  4. (Why) do we need to call cache or persist on a RDD
  5. Automatically and Elegantly flatten DataFrame in Spark SQL
  6. How do I skip a header from CSV files in Spark?
  7. Spark extracting values from a Row
  8. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  9. Provide schema while reading csv file as a dataframe
  10. How to save DataFrame directly to Hive?
  11. About how to add a new column to an existing DataFrame with random values in Scala
  12. Joining Spark dataframes on the key
  13. How to use Column.isin with list?
  14. Modify collection inside a Spark RDD foreach
  15. Function returns an empty List in Spark
  16. How to write unit tests in Spark 2.0+?
  17. How to compare two dataframe and print columns that are different in scala
  18. Operate on neighbor elements in RDD in Spark
  19. Customize SparkContext using sparkConf.set(..) when using spark-shell
  20. Renaming column names of a DataFrame in Spark Scala
  21. What is going wrong with `unionAll` of Spark `DataFrame`?
  22. Spark: Transpose DataFrame Without Aggregating
  23. Exploding nested Struct in Spark dataframe
  24. Spark: long delay between jobs
  25. How to get keys and values from MapType column in SparkSQL DataFrame
  26. Addition of two RDD[mllib.linalg.Vector]’s
  27. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  28. How to create DataFrame from Scala’s List of Iterables?
  29. How to create a Dataset of Maps?
  30. Spark ML VectorAssembler returns strange output

Leave a Comment