How to turn off INFO logging in Spark?

by

Just execute this command in the spark directory:

cp conf/log4j.properties.template conf/log4j.properties

Edit log4j.properties:

# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

Replace at the first line:

log4j.rootCategory=INFO, console

by:

log4j.rootCategory=WARN, console

Save and restart your shell. It works for me for Spark 1.1.0 and Spark 1.5.1 on OS X.

More Related Contents:

  1. Calling Java/Scala function from a task
  2. How to use JDBC source to write and read data in (Py)Spark?
  3. How to use a Scala class inside Pyspark
  4. What is the equivalent to scala.util.Try in pyspark?
  5. How to split Vector into columns – using PySpark
  6. Spark: How to map Python with Scala or Java User Defined Functions?
  7. Pyspark: Split multiple array columns into rows
  8. How to change a dataframe column from String type to Double type in PySpark?
  9. importing pyspark in python shell
  10. How can PySpark be called in debug mode?
  11. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  12. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  13. How to add third-party Java JAR files for use in PySpark
  14. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  15. Filter Pyspark dataframe column with None value
  16. PySpark logging from the executor
  17. Convert spark DataFrame column to python list
  18. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  19. Load CSV file with PySpark
  20. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  21. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  22. ImportError: No module named numpy on spark workers
  23. How to pass a constant value to Python UDF?
  24. How to zip two array columns in Spark SQL
  25. How to run stored procedure on SQL server from Spark (Databricks) JDBC python?
  26. Multiple condition filter on dataframe
  27. How to return a “Tuple type” in a UDF in PySpark?
  28. How to connect HBase and Spark using Python?
  29. Pyspark 2.4.0, read avro from kafka with read stream – Python
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment