ClassPath: ClassPath is affected depending on what you provide. There are a couple of ways to set something on the classpath: spark.driver.extraClassPath or it’s alias –driver-class-path to set extra classpaths on the node running the driver. spark.executor.extraClassPath to set extra class path on the Worker nodes. If you want a certain JAR to be effected … Read more
Edit your conf/log4j.properties file and change the following line: log4j.rootCategory=INFO, console to log4j.rootCategory=ERROR, console Another approach would be to : Start spark-shell and type in the following: import org.apache.log4j.Logger import org.apache.log4j.Level Logger.getLogger(“org”).setLevel(Level.OFF) Logger.getLogger(“akka”).setLevel(Level.OFF) You won’t see any logs after that. Other options for Level include: all, debug, error, fatal, info, off, trace, trace_int, warn Details … Read more