Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?

by

In this way it worked easily.. instead of specifying each jar with version separately..

#!/bin/sh
# build all other dependent jars in OTHER_JARS

JARS=`find ../lib -name '*.jar'`
OTHER_JARS=""
   for eachjarinlib in $JARS ; do    
if [ "$eachjarinlib" != "APPLICATIONJARTOBEADDEDSEPERATELY.JAR" ]; then
       OTHER_JARS=$eachjarinlib,$OTHER_JARS
fi
done
echo ---final list of jars are : $OTHER_JARS
echo $CLASSPATH

spark-submit --verbose --class <yourclass>
... OTHER OPTIONS
--jars $OTHER_JARS,APPLICATIONJARTOBEADDEDSEPERATELY.JAR

  • Using tr unix command also can help like the below example.

    --jars $(echo /dir_of_jars/*.jar | tr ' ' ',')

More Related Contents:

  1. Add JAR files to a Spark job – spark-submit
  2. Resolving dependency problems in Apache Spark
  3. Spark: How to map Python with Scala or Java User Defined Functions?
  4. Spark read file from S3 using sc.textFile (“s3n://…)
  5. Spark Strutured Streaming automatically converts timestamp to local time
  6. How can I update a broadcast variable in spark streaming?
  7. Apache Spark – foreach Vs foreachPartition When to use What?
  8. Matrix Multiplication in Apache Spark [closed]
  9. What are the Spark transformations that causes a Shuffle?
  10. Is gzip format supported in Spark?
  11. Change output filename prefix for DataFrame.write()
  12. How can I force Spark to execute code?
  13. How to save models from ML Pipeline to S3 or HDFS?
  14. Creating a jar file from a Scala file
  15. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  16. How to list the files inside a JAR file?
  17. Run jar file in command prompt [duplicate]
  18. How to convert AAR to JAR
  19. Eclipse error: “The import XXX cannot be resolved”
  20. How to read a file from jar in Java?
  21. Extract and load DLL from JAR
  22. Shuffle a list of integers with Java 8 Streams API
  23. Python: How can I execute a jar file through a python script
  24. Creating Runnable Jar with external files included
  25. Classpath resource within jar
  26. How do I call a UDF on a Spark DataFrame using JAVA?
  27. Eclipse Java; export jar, include referenced libraries, without fatjar
  28. Double Clicking JAR file does not open Command Prompt
  29. How to create my own java library(API)?
  30. How to make Eclipse see the changes in Play! compiled templates?

Leave a Comment