How to log using log4j to local file system inside a Spark application that runs on YARN?

by

It looks like you’ll need to append to the JVM arguments used when launching your tasks/jobs.

Try editing conf/spark-defaults.conf as described here

spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j.properties

spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j.properties

Alternatively try editing conf/spark-env.sh as described here to add the same JVM argument, although the entries in conf/spark-defaults.conf should work.

If you are still not getting any joy, you can explicitly pass the location of your log4j.properties file on the command line along with your spark-submit like this if the file is contained within your JAR file and in the root directory of your classpath

spark-submit --class sparky.MyApp --master spark://my.host.com:7077 --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j-executor.properties" myapp.jar

If the file is not on your classpath use the file: prefix and full path like this

spark-submit ... --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j-executor.properties" ...

More Related Contents:

  1. log4j logging hierarchy order
  2. Where are logs in Spark on YARN?
  3. How to Create a Custom Appender in log4j2?
  4. How to log cron jobs?
  5. How can I create 2 separate log files with one log4j config file?
  6. log4j: Log output of a specific class to a specific appender
  7. Configuring Hibernate logging using Log4j XML config file?
  8. Create an MSI log file
  9. log4j configuration via JVM argument(s)?
  10. How to view the logging of Google Web App when other users are executing it?
  11. Best way to log POST data in Apache?
  12. Karate: Is there a way to disable log when using retry?
  13. Logging POST data from $request_body
  14. Spark yarn cluster vs client – how to choose which one to use?
  15. PySpark logging from the executor
  16. How to do logging in React Native
  17. how to make log4j to write to the console as well
  18. How to delete old logs with log4j2
  19. Redirect System.out and System.err to slf4j
  20. How to specify Log4J 2.x config location?
  21. Save Screen (program) output to a file
  22. Which cluster type should I choose for Spark? [closed]
  23. Configuring RollingFileAppender in log4j
  24. Parse a log4j log file
  25. How do I get logs from all pods of a Kubernetes replication controller?
  26. How to fix JSP compiler warning: one JAR was scanned for TLDs yet contained no TLDs?
  27. Log4J2 property substitution – default
  28. Spark: get number of cluster cores programmatically
  29. How to create different log files for different packages using same log4j logger?
  30. Log4j2 (2.1) custom plugin not detected by packages attribute

Leave a Comment